Once I figured out perl "cheats" during "line processing" operations (perl -ne 's/a/b/g && print') by gobbling big chunks of input into memory and started to emulate this in nL, string processing speeds became equal.
This made me think of serving web pages. nL can (a) work as a standalone server for smaller applications (b) can be spawned in scores of copies because it's so light and/or (c) be dropped into a cgi-bin directory.
So as it is it's already more versatile than other languages.
newLisp, however, is unlike the "big" ones in the sense that there is no immediate way to embed it into a web server, in the manner of mod_perl or the php apache module.
2. an interesting find - and how it works
Thinking of that, I looked around and made an interesting discovery. Some lisp people several years ago (development seems to have stopped in the first half of 2005) created "mod_lisp" for Apache 2.0.x
In fact, it's much better than a lisp-specific module. It's totally generic, and any language with networking scripting capabilities could make use of it.
In fact, there is one well-known protocol that works exactly as described - FastCGI. The difference is that FCGI's protocol is not ASCII, so scripting for it would be a much bigger headache. FCGI is described in an RFC full of unbelievably bureaucratic lingo, etc. - but it's the same idea.What the module does is make Apache, when a request comes for a URL configured to be taken care of by the module (e.g. to www.server.com/lisp/xxx.html), try to connect to a network server at its back (written in lisp, or in any other language) and running either locally, or across a network on a backend machine.
Knocking on a preconfigured port (say, 3000), the mod_lisp from Apache will send all typical web environment and data using a very simple ASCII protocol. The kind of data we know well from playing with CGI scripts.
Then the our Listener, the Lisp server, will decipher the request and send back the header and content, again, very much the way they do over regular CGI. The difference is that there is no need any longer to start a new process with all its overhead. One binary serving the connection will just loop and send the pages.
3. what i did as an experiment
As a result of this sudden fit of hacking enthusiasm I spend last Sunday trying to make mod_lisp work in the hope that nL won't let me down and compare well against 6 responses per second from CMUCL (or was it CLISP?) when serving from a MySQL database via mod_lisp, or the whopping 36/sec when the database was detached -- /figures corrected - lithper/. That's what authors of the module achieved with their big Lisps according to their measurements back in 2003.
(my test box is a 500MHz from 2000, so I would not have unjust advantage, I thought).
[.a.]. mod_lisp did not compile with current apache 2.2 In series 2.2 APR (apache API's to its libraries) changed to a new version.
Previous 2.0 is still maintained (the latest release 2.0.63 fixes a number of cross-scripting vulnerabilities), as is the old 1.3 series (latest with security fixes etc is 1.3.41)
I caught several obvious things that needed updating when attempting to port mod_lisp to 2.2, but finally, although the module compiled and passed apache checks, ver2.2.8 segfaulted with the module present. Well..
[.b.]. Then I downloaded apache 2.0.53 from 2005, figuring that the module was still alive at that time to bypass possible incompatibilities.
Now it compiled and installed itself OK
[.c.] Next I had to spend a few hours trying to understand from a very brief description the correct syntax for the protocol, reading CL lisp source of the provided samples, and experimenting with a basic setup of nL network server.
In the end, the protocol looks very simple indeed, one just needs to know what it chokes on.
So finally for a first estimation I made nL serve a 10kB web page (pure text) as a server sitting behind apache and talking to it through a socket -- in comparison to the same text being served via a light cgi script. (it basically checks the arguments, and then copies the file picking it from the filesystem; in both cases I just "cat" the file in response to a request)
4. The results as measured by "ab"
(apache bench, a small utility in apache distrbution). are as follows:
(CGI app) - nL is was light enough to serve at the rate of 50/second. It is OK - many commercial sites with LAMP architectures can achieve 6-10 in their best days, while under the load a user will have to wait for 2-3, even 5 seconds before a page appears.
(mod_lisp), a "fastCGI" for the nerd who assembles his helicopter from parts every time before he goes to fly (and calls it "the unix way"), however, somewhat surprised me. The page was delivered at consistent 250 times/second. (Of course, in real life the script will nave to do more than just spew a small file out, but..)
This is better than mod_perl, i.e. perl embedded into apache does on my machine (or at least it's the same speed; my fully enabled web app is a bit heavier, and I remember under mod_perl it ran at 80-100 responses/second on the same type of output files).
5. The problem is that the setup at this point is buggy. nLisp can fall into coma, frequently, and refuse to serve. Possibly the barebone script was not correct, and I would need to set up some request queue, or maybe it's the problem of consistency across several copies of the web server, or maybe the "fastCGI" scripts must be written in a special way, as my past experience with mod_perl and such suggests.
Update: it seems apache closed connections, so that newLisp server would try to write data nowhere, when apache limits on the number of processes and clients were not sufficient.
It seems the setup works.
NewLISP script should simply reset connection when this happens (rarely if at all) and keep listening from its standard server loop-function forever.
NewLisp serves all of the stuff amazingly fast. The totals confirm the volume is correct (i.e. no 0-length pages or error pages in the output, it seems to add up).
If this message interests readers, I'll add a brief how-to and links.First Impression
My first impression is that mod_lisp, a generic simpler clone of fastCGI, should be supported and ported to the later apache releases.
And that proper server-side script should be written for nL to work in this mode. (upd: actually, the simplest standard newLisp server handles the communication OK)
It adds to the already wider set of ways to serve web pages than with other languages, and mod_lisp web interfaces in nL might prove to be a unique performer fit for higher capacity sites, both commercial or not.
Alternatively, it might be worth spending some effort on writing a FastCGI protocol server to drop into nL scripts in view of the encouraging results given by the mod_lisp ascii clone. FastCGI is supported by a large number of web servers, mega- or nano-sized, in contrast to native embedding like apache embeds perl and php
(one othe approach would be to import symbols from the reference implementation of Fast CGI C library; it would break the "standaloneness" of nL script, however, if that's important)
Below are two sample reports from "ab".
CGI run was 15kB - against 9.5-10kB from mod_lisp, but the rate per second was pretty much constant and consistent for the two setups.
I used the same web server in both cases, so its configuration supposedly affected both results in a similar way. It had no specific caching enabled. The test is, of course, purely informal.
1. 300 requests with concurrency of 100 - i.e. 3 seconds if the server can bear such load ideally, or longer, if it backlogs under the load. This test bears the load, but delays responses about 2 seconds already:
2. same 300/100 for mod_lispuser@host-> ./ab -n 300 -c 100 'http://localhost/cgi-bin/scriptname.lsp ... ser=nick&f
ilename=asdf'
This is ApacheBench, Version 2.0.40-dev .............
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Finished 300 requests
Server Software: Apache/2.0.49
Server Hostname: localhost
Server Port: 80
Document Path: /cgi-bin/script.lsp?target=edit_file&arch_path=local-blogs&rss_user=nick&filename=asdf
Document Length: 15130 bytes
Concurrency Level: 100
Time taken for tests: 5.974378 seconds
Complete requests: 300
Failed requests: 0
Write errors: 0
Total transferred: 4585956 bytes
HTML transferred: 4539000 bytes
Requests per second: 50.21 [#/sec] (mean)
Time per request: 1991.459 [ms] (mean)
Time per request: 19.915 [ms] (mean, across all concurrent requests)
Transfer rate: 749.53 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 10.8 0 37
Processing: 51 1687 569.4 1915 2928
Waiting: 47 1669 559.9 1912 2608
Total: 88 1694 558.9 1915 2928
Percentage of the requests served within a certain time (ms)
50% 1915
66% 1961
75% 2003
80% 2028
90% 2079
95% 2157
98% 2441
99% 2610
100% 2928 (longest request)
user@host ->
This test bore the load easily, with 0.3-0.4 seconds per response, and that at 250 requests per second. It means the server could sustain larger loads and still provide usable delays
.user@host -> ./ab -n 300 -c 100 'http://localhost:80/lisp/index.html'
This is ApacheBench, Version 2.0.40-dev ........
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Finished 300 requests
Server Software: Apache/2.0.49
Server Hostname: localhost
Server Port: 80
Document Path: /lisp/index.html
Document Length: 9444 bytes
Concurrency Level: 100
Time taken for tests: 1.160991 seconds
Complete requests: 300
Failed requests: 0
Write errors: 0
Total transferred: 2893500 bytes
HTML transferred: 2833200 bytes
Requests per second: 258.40 [#/sec] (mean)
Time per request: 386.997 [ms] (mean)
Time per request: 3.870 [ms] (mean, across all concurrent requests)
Transfer rate: 2433.27 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 12.3 0 42
Processing: 3 316 107.7 371 399
Waiting: 0 316 107.6 369 398
Total: 45 323 95.5 371 399
Percentage of the requests served within a certain time (ms)
50% 371
66% 384
75% 385
80% 385
90% 389
95% 389
98% 390
99% 392
100% 399 (longest request)
user@host ->