CGI input data disappearing
CGI input data disappearing
Hi, all,
I've had an issue that bothered me for a while now. Several times I've noticed that data, which newLISP receives from CGI, is incomplete. I don't know if it's a newLISP issue or not, but I've experienced it with CGI under both Apache (behind Squid on NearlyFreeSpeech) and mathopd (behind nginx).
Please see http://km.krot.org/cgitest.cgi for demonstration (source here: http://km.krot.org/cgitest.txt)
The scripts create a string with 20000 characters. Then the script generates a page, where the string is sent as a hidden parameter and textarea. Then the script compares the lengths. What worries me a lot is that the results are varying between form submissions.
Does anyone have any clues?
Kirill
I've had an issue that bothered me for a while now. Several times I've noticed that data, which newLISP receives from CGI, is incomplete. I don't know if it's a newLISP issue or not, but I've experienced it with CGI under both Apache (behind Squid on NearlyFreeSpeech) and mathopd (behind nginx).
Please see http://km.krot.org/cgitest.cgi for demonstration (source here: http://km.krot.org/cgitest.txt)
The scripts create a string with 20000 characters. Then the script generates a page, where the string is sent as a hidden parameter and textarea. Then the script compares the lengths. What worries me a lot is that the results are varying between form submissions.
Does anyone have any clues?
Kirill
Re: CGI input data disappearing
I suspect the issue is with newLISP, as perl works just fine:
http://km.krot.org/cgitest2.cgi (source: http://km.krot.org/cgitest2.txt).
All clues welcome!
Kirill
http://km.krot.org/cgitest2.cgi (source: http://km.krot.org/cgitest2.txt).
All clues welcome!
Kirill
Re: CGI input data disappearing
This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html
ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67
ps: updated to 3.11 cgi.lsp a few minutes later
ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67
ps: updated to 3.11 cgi.lsp a few minutes later
Re: CGI input data disappearing
Who is Rocket Man?
And why does he/she use tabs to indent newLISP code? ;)
And why does he/she use tabs to indent newLISP code? ;)
(λx. x x) (λx. x x)
Re: CGI input data disappearing
Thanks, but now CGI:get returns nothing at all: http://km.krot.org/cgitest3.cgiLutz wrote:This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html
ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67
ps: updated to 3.11 cgi.lsp a few minutes later
Re: CGI input data disappearing
Patch for cgi.lsp: http://km.krot.org/code/cgi.lsp.patch
Now http://km.krot.org/cgitest3.cgi works properly as well.
Now http://km.krot.org/cgitest3.cgi works properly as well.
Re: CGI input data disappearing
When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.
With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
Re: CGI input data disappearing
And here's a patch for web.lsp: http://km.krot.org/code/web.lsp.patch
Re: CGI input data disappearing
True, but I'm using their beta "realm", so newLISP and other tools are not so very old. newLISP version there isLutz wrote:When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.
Code: Select all
newLISP v.10.3.3 on BSD IPv4/6 UTF-8.
Yes, that's why the patched code reads until it has read enough.Lutz wrote:With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
Re: CGI input data disappearing
Thanks to Kirill, there's now a patch that also fixes this problem for web.lsp (part of artful-code).
You can always grab the latest version at:
https://github.com/kanendosei/artful-newlisp
You can always grab the latest version at:
https://github.com/kanendosei/artful-newlisp
. Kanen Flowers http://kanen.me .
Re: CGI input data disappearing
The issue is solved, so I remove the test CGI scripts. Thanks.
Re: CGI input data disappearing
cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html
Re: CGI input data disappearing
Thanks, Lutz! The file online does not seem to include the change below, thoughLutz wrote:cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html
Code: Select all
--- cgi.lsp.orig 2012-12-05 00:48:29.632208151 +0000
+++ cgi.lsp 2012-12-05 00:51:34.799112862 +0000
@@ -132,10 +132,13 @@
; get POST data if present, use CONTENT_LENGTH variable
; if available
(if (env "CONTENT_LENGTH")
- (set 'post-data "")
(when (= (env "REQUEST_METHOD") "POST")
- (while (read (device) buffer (int (env "CONTENT_LENGTH")))
- (write post-data buffer))
+ (set 'recvd 0)
+ (set 'conln (int (env "CONTENT_LENGTH")))
+ (set 'post-data "")
+ (do-while (< recvd conln)
+ (inc recvd (read (device) buffer conln))
+ (write post-data buffer))
(let (boundary (when (regex "multipart/form-data; boundary=(.*)"
(env "CONTENT_TYPE")) (append "--" $1)))
Re: CGI input data disappearing
Two things:
(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.
currently in cgi.lsp 3.2:
(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.
(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.
currently in cgi.lsp 3.2:
Code: Select all
(if (env "CONTENT_LENGTH")
(when (= (env "REQUEST_METHOD") "POST")
(set 'post-data "")
(while (read (device) buffer (int (env "CONTENT_LENGTH")))
(write post-data buffer))
Re: CGI input data disappearing
Thank you, Lutz.
Yes, the diff was against 3.11, which did not work, and I did not notice that the said statement was moved from before 'when' to after 'when'.Lutz wrote:Two things:
(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.
Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?Lutz wrote:(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.
Re: CGI input data disappearing
Yes, perhaps. But then also a response or error page should be formatted and sent back to the client. I guess in the real world, CONTENT_LENGTH will be ok most of the time and its not an issue to worry about. Almost always, CONTENT_LENGTH will be generated automatically by software.Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?
I wonder how other web frameworks deal with these cases.