Uploading files problem

[TPJ]

I have a web app written in newLISP. The app is served by the newLISP interpreter run in the HTTP server mode. The server runs behind a proxy, which is nginx. So the flow is simple: nginx gets the requests, and proxies it to the newLISP HTTP server.

I have problems with files uploading. The script mentioned in this topic doesn't work, since no CONTENT_LENGTH (or CONTENT_TYPE) environment variable is set (I checked it). BTW, I used the upload.html code to perform all tests.

I tried to just read the stdin, but it doesn't work. My CGI scripts (pure newLISP) get nothing to read (stdin is empty).

When I tried to reach the newLISP HTTP server directly (without nginx serving as a proxy), I was able to upload a text file. I just used a simple script, that reads stdin, and gets all the content of the sent file:

Code: Select all

(set 'infile (open "upload-file" "write"))
(while (read (device) buffer 1024)
  (write infile buffer))
(close infile)

Unfortunatelly, sending any binary file (e.g. images) doesn't work at all. The stdin is broken after a few bytes read. For example, in case of a GIF image, all I got was:

Code: Select all

-----------------------------1124875958105779246987104586
Content-Disposition: form-data; name="uploaded_data"; filename="polsl.gif"
Content-Type: image/gif

GIF89a_

Now, the questions.

1) Why on earth my script doesn't get anything on stdin when newLISP HTTP server is run behind a proxy (nginx in my case)?
2) Why on earth binary files aren't properly sent, while text files are?

I'll appreciate any help.

[Lutz]

I am not familiar with the nginx proxy server, but when using the Apache HTTP server, use this:

http://www.newlisp.org/index.cgi?page=F ... oad_Script

also referenced from here:

http://www.newlisp.org/index.cgi?Tips_and_Tricks

The server side upload script handles a POST request from a client HTML form.

When uploading a file to a newLISP HTTP server directly without a proxy in-between, you can use '(write-file <target URL> <content>)' on the client directly, no script is necessary server side as newLISP HTTP server has GET/PUT/POST/DELETE handling built-in (only POST requests would need a script handler):

Code: Select all

(write-file "http://thesite.com/binaryfile.bin" (read-file "binarycontent"))

'write-file' when seeing a 'http://' prefixed target will issue a HTTP PUT request to newLISP HTTP server and it works with binary content on Windows and all Unix platforms.

[TPJ]

Thanks for your remarks. Using write-file is not a solution, since I'm working on a web application. My only option is to use the POST method.


I'll start with a little update. I've been trying to solve that problem for two days now (spending on it my full time), and I've made some interesting observations:

1) Reading the stdin from a newLISP CGI script works perfectly, as long as a text file is uploaded. It works perfectly in both cases: when using nginx as a proxy, and without any proxy server.

2) Reading the stdin from a newLISP CGI script fails when a binary file is uploaded. After reading a few bytes, the stdin suddenly breakes. Once again: it happens in both cases: with ngxinx proxy, and without it.

I have described this problem here, on stackoverflow (you can check there what exactly I get from stdin in my scripts). The proposed solution didn't help, though. But let's forget about this problem, for a while.

I decided to investigate it more. I've installed the thttpd server, and used it to perform the same task: serve two pages, one upload.html, and one CGI script, that should just read the stdin and save it to some file. I used a simple shell script to do the job:

Code: Select all

#!/bin/sh

cat > to-file

echo -e "Content-type: text/html\r\n\r\n"
echo "<p>Hello!</p>"

The result: everything ran fine, just as it should. I was able to upload any file I wanted.

So, I tried to do the same thing using the newLISP HTTP server, and exactly the same two pages (including the simple shell script). The result: all binary files uploaded were broken, just as in the first case I described in this post.

I'm not an HTTP server expert... Therefore I hope that my conclusion is wrong. But the conclusion is that something is wrong with the newLISP HTTP server - since it's the only thing that had been changed before everything stopped to work correctly. And I don't like this conclusion, since I intend to use the newLISP HTTP server a lot.

I tested it all on my Arch Linux box, if it is of any importance.

[Lutz]

Currently newLISP HTTP server only handles textual data in POST requests and doesn't handle multipart/form-data content-type at all because the CONTENT_TYPE environment variable is not set. This has now been added to 10.2.17:

http://www.newlisp.org/downloads/develo ... nprogress/

and it works well with the upload.cgi script and binary content:

http://www.newlisp.org/index.cgi?page=F ... oad_Script

tested on Mac OS X 10.6.4 and UBUNTU Linux 10.4