CGI input data disappearing

Machine-specific discussion
Unix, Linux, OS X, OS/2, Windows, ..?
Locked
Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

CGI input data disappearing

Post by Kirill »

Hi, all,

I've had an issue that bothered me for a while now. Several times I've noticed that data, which newLISP receives from CGI, is incomplete. I don't know if it's a newLISP issue or not, but I've experienced it with CGI under both Apache (behind Squid on NearlyFreeSpeech) and mathopd (behind nginx).

Please see http://km.krot.org/cgitest.cgi for demonstration (source here: http://km.krot.org/cgitest.txt)

The scripts create a string with 20000 characters. Then the script generates a page, where the string is sent as a hidden parameter and textarea. Then the script compares the lengths. What worries me a lot is that the results are varying between form submissions.

Does anyone have any clues?

Kirill

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

I suspect the issue is with newLISP, as perl works just fine:

http://km.krot.org/cgitest2.cgi (source: http://km.krot.org/cgitest2.txt).

All clues welcome!

Kirill

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: CGI input data disappearing

Post by Lutz »

This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html


ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67

ps: updated to 3.11 cgi.lsp a few minutes later

rickyboy
Posts: 607
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: CGI input data disappearing

Post by rickyboy »

Who is Rocket Man?

And why does he/she use tabs to indent newLISP code? ;)
(λx. x x) (λx. x x)

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

Lutz wrote:This updated version 3.1 of cgi.lsp should fix this: http://www.newlisp.org/code/modules/cgi.lsp.html


ps: see also here http://newlisponrockets.com/rockets-item.lsp?p=67

ps: updated to 3.11 cgi.lsp a few minutes later
Thanks, but now CGI:get returns nothing at all: http://km.krot.org/cgitest3.cgi

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

I have a fix! http://km.krot.org/cgitest4.cgi

Patch will follow shortly...

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

Patch for cgi.lsp: http://km.krot.org/code/cgi.lsp.patch

Now http://km.krot.org/cgitest3.cgi works properly as well.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: CGI input data disappearing

Post by Lutz »

When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.

With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

And here's a patch for web.lsp: http://km.krot.org/code/web.lsp.patch

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

Lutz wrote:When running on nfshost.com, make sure you do not run with /usr/local/bin/newlisp, which probably is a very old version of newLISP. Instead compile your own version of newLISP on nfshost and put it in e.g: /home/public/bin/newlisp, and of course put #!/home/public/bin/newlisp as the first line in your cgi files.
True, but I'm using their beta "realm", so newLISP and other tools are not so very old. newLISP version there is

Code: Select all

newLISP v.10.3.3 on BSD IPv4/6 UTF-8.
Lutz wrote:With the old cgi.lsp 3.0, it failed me about one in 10 times on Apache. With the new cgi.lsp 3.11, I did more than 100 invocations and has not failed yet. The problem probably occurred when large POST data was received in more than one buffer with less bytes than in CONTENT_LENGTH.
Yes, that's why the patched code reads until it has read enough.

kanen
Posts: 145
Joined: Thu Mar 25, 2010 6:24 pm
Contact:

Re: CGI input data disappearing

Post by kanen »

Thanks to Kirill, there's now a patch that also fixes this problem for web.lsp (part of artful-code).

You can always grab the latest version at:

https://github.com/kanendosei/artful-newlisp
. Kanen Flowers http://kanen.me .

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

The issue is solved, so I remove the test CGI scripts. Thanks.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: CGI input data disappearing

Post by Lutz »

cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

Lutz wrote:cgi.lsp has been updated again, to version 3.2 : http://www.newlisp.org/code/modules/cgi.lsp.html
Thanks, Lutz! The file online does not seem to include the change below, though

Code: Select all

--- cgi.lsp.orig	2012-12-05 00:48:29.632208151 +0000
+++ cgi.lsp	2012-12-05 00:51:34.799112862 +0000
@@ -132,10 +132,13 @@
 ; get POST data if present, use CONTENT_LENGTH variable
 ; if available
 (if (env "CONTENT_LENGTH")
-    (set 'post-data "")
     (when (= (env "REQUEST_METHOD") "POST")
-        (while (read (device) buffer (int (env "CONTENT_LENGTH")))
-            (write post-data buffer))
+         (set 'recvd 0)
+         (set 'conln (int (env "CONTENT_LENGTH")))
+         (set 'post-data "")
+         (do-while (< recvd conln)
+                   (inc recvd (read (device) buffer conln))
+                   (write post-data buffer))
 
         (let (boundary (when (regex "multipart/form-data; boundary=(.*)"
                 (env "CONTENT_TYPE")) (append "--" $1)))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: CGI input data disappearing

Post by Lutz »

Two things:

(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.

currently in cgi.lsp 3.2:

Code: Select all

(if (env "CONTENT_LENGTH")
    (when (= (env "REQUEST_METHOD") "POST")
        (set 'post-data "")
        (while (read (device) buffer (int (env "CONTENT_LENGTH")))
            (write post-data buffer))
(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Re: CGI input data disappearing

Post by Kirill »

Thank you, Lutz.
Lutz wrote:Two things:

(1) I believe the version you diffed against is not the current 3.2 online - may be you have to hit refresh in your browser. See the position of the (set 'post-data "") statement, which has to come after the 'when'. This was the change from 3.11 to 3.2.
Yes, the diff was against 3.11, which did not work, and I did not notice that the said statement was moved from before 'when' to after 'when'.
Lutz wrote:(2) When less data are in the channel than announced in CONTENT_LENGTH, the web.lsp version will throw an error when the 'inc' is fed a 'nil' increment. The cgi.lsp will simply stop reading. When CONTENT_LENGTH is too small cgi.lsp will continue reading. Both versions will work with correctly formatted requests, but cgi.lsp is more robust when less bytes are available than in CONTENT_LENGTH.
Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: CGI input data disappearing

Post by Lutz »

Maybe it is good to throw error when CONTENT_LENGTH does not correspond to available data?
Yes, perhaps. But then also a response or error page should be formatted and sent back to the client. I guess in the real world, CONTENT_LENGTH will be ok most of the time and its not an issue to worry about. Almost always, CONTENT_LENGTH will be generated automatically by software.

I wonder how other web frameworks deal with these cases.

Locked