Page 1 of 1
CGI input data disappearing
Posted: Tue Dec 04, 2012 11:28 am
by Kirill
Hi, all,
I've had an issue that bothered me for a while now. Several times I've noticed that data, which newLISP receives from CGI, is incomplete. I don't know if it's a newLISP issue or not, but I've experienced it with CGI under both Apache (behind Squid on NearlyFreeSpeech) and mathopd (behind nginx).
Please see
http://km.krot.org/cgitest.cgi for demonstration (source here:
http://km.krot.org/cgitest.txt)
The scripts create a string with 20000 characters. Then the script generates a page, where the string is sent as a hidden parameter and textarea. Then the script compares the lengths. What worries me a lot is that the results are varying between form submissions.
Does anyone have any clues?
Kirill
Re: CGI input data disappearing
Posted: Tue Dec 04, 2012 12:46 pm
by Kirill
I suspect the issue is with newLISP, as perl works just fine:
http://km.krot.org/cgitest2.cgi (source:
http://km.krot.org/cgitest2.txt).
All clues welcome!
Kirill
Re: CGI input data disappearing
Posted: Tue Dec 04, 2012 8:25 pm
by Lutz
This updated version 3.1 of cgi.lsp should fix this:
http://www.newlisp.org/code/modules/cgi.lsp.html
ps: see also here
http://newlisponrockets.com/rockets-item.lsp?p=67
ps: updated to 3.11 cgi.lsp a few minutes later
Re: CGI input data disappearing
Posted: Tue Dec 04, 2012 11:29 pm
by rickyboy
Who is Rocket Man?
And why does he/she use tabs to indent newLISP code? ;)
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 12:27 am
by Kirill
Thanks, but now CGI:get returns nothing at all:
http://km.krot.org/cgitest3.cgi
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 12:47 am
by Kirill
I have a fix!
http://km.krot.org/cgitest4.cgi
Patch will follow shortly...
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 12:52 am
by Kirill
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 1:09 am
by Lutz
When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.
With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 1:14 am
by Kirill
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 1:17 am
by Kirill
Lutz wrote:When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.
True, but I'm using their beta "realm", so newLISP and other tools are not so very old. newLISP version there is
Code: Select all
newLISP v.10.3.3 on BSD IPv4/6 UTF-8.
Lutz wrote:With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
Yes, that's why the patched code reads until it has read enough.
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 1:25 am
by kanen
Thanks to Kirill, there's now a patch that also fixes this problem for web.lsp (part of artful-code).
You can always grab the latest version at:
https://github.com/kanendosei/artful-newlisp
Re: CGI input data disappearing
Posted: Wed Dec 05, 2012 8:56 am
by Kirill
The issue is solved, so I remove the test CGI scripts. Thanks.
Re: CGI input data disappearing
Posted: Thu Dec 06, 2012 5:32 am
by Lutz
cgi.lsp has been updated again, to version 3.2 :
http://www.newlisp.org/code/modules/cgi.lsp.html
Re: CGI input data disappearing
Posted: Thu Dec 06, 2012 12:44 pm
by Kirill
Thanks, Lutz! The file online does not seem to include the change below, though
Code: Select all
--- cgi.lsp.orig 2012-12-05 00:48:29.632208151 +0000
+++ cgi.lsp 2012-12-05 00:51:34.799112862 +0000
@@ -132,10 +132,13 @@
; get POST data if present, use CONTENT_LENGTH variable
; if available
(if (env "CONTENT_LENGTH")
- (set 'post-data "")
(when (= (env "REQUEST_METHOD") "POST")
- (while (read (device) buffer (int (env "CONTENT_LENGTH")))
- (write post-data buffer))
+ (set 'recvd 0)
+ (set 'conln (int (env "CONTENT_LENGTH")))
+ (set 'post-data "")
+ (do-while (< recvd conln)
+ (inc recvd (read (device) buffer conln))
+ (write post-data buffer))
(let (boundary (when (regex "multipart/form-data; boundary=(.*)"
(env "CONTENT_TYPE")) (append "--" $1)))
Re: CGI input data disappearing
Posted: Thu Dec 06, 2012 2:45 pm
by Lutz
Two things:
(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the
(set 'post-data "") statement, which has to come
after the 'when'. This was the change from 3.11 to 3.2.
currently in cgi.lsp 3.2:
Code: Select all
(if (env "CONTENT_LENGTH")
(when (= (env "REQUEST_METHOD") "POST")
(set 'post-data "")
(while (read (device) buffer (int (env "CONTENT_LENGTH")))
(write post-data buffer))
(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.
Re: CGI input data disappearing
Posted: Thu Dec 06, 2012 3:51 pm
by Kirill
Thank you, Lutz.
Lutz wrote:Two things:
(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.
Yes, the diff was against 3.11, which did not work, and I did not notice that the said statement was moved from before 'when' to after 'when'.
Lutz wrote:(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.
Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?
Re: CGI input data disappearing
Posted: Thu Dec 06, 2012 4:24 pm
by Lutz
Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?
Yes, perhaps. But then also a response or error page should be formatted and sent back to the client. I guess in the real world, CONTENT_LENGTH will be ok most of the time and its not an issue to worry about. Almost always, CONTENT_LENGTH will be generated automatically by software.
I wonder how other web frameworks deal with these cases.