get-url of empty content fails (fixed in 10.6.4)

[hartrock]

A GET resulting into a reply with Content-Length 0 should be possible: from
http://svn.tools.ietf.org/svn/wg/httpbi ... ssage.body
:

The presence of a message body in a request is signaled by a Content-Length or Transfer-Encoding header field. ...

... All 1xx (Informational), 204 (No Content), and 304 (Not Modified) responses do not include a message body. All other responses do include a message body, although the body might be of zero length.

So this applies to e.g. '200 OK' and '404 Not Found' responses, which may have a zero message body.

newLISP:

Code: Select all

sr@mad:~$ newlisp
newLISP v.10.6.2 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (get-url "http://localhost/empty.txt" "debug")
GET /empty.txt HTTP/1.1
Host: localhost
User-Agent: newLISP v10602
Connection: close

HTTP/1.1 200 OK

"ERR: HTTP document empty"
> ; but:
> (get-url "file:///tmp/empty.txt" "debug")
""


curl:

Code: Select all

sr@mad:~$ curl -v -X GET http://localhost/empty.txt
* About to connect() to localhost port 80 (#0)
*   Trying ::1...
* connected
* Connected to localhost (::1) port 80 (#0)
> GET /empty.txt HTTP/1.1
> User-Agent: curl/7.26.0
> Host: localhost
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Date: Fri, 10 Jul 2015 12:49:10 GMT
< Server: Apache/2.2.22 (Debian)
< Last-Modified: Fri, 10 Jul 2015 12:10:49 GMT
< ETag: "8075b-0-51a84433bdbcf"
< Accept-Ranges: bytes
< Content-Length: 0
< Vary: Accept-Encoding
< Content-Type: text/plain
<
* Connection #0 to host localhost left intact
* Closing connection #0


(empty message body).

Ideas:

• (http-error) for more fine-granular checking of http responses?
• net-error '24 HTTP document empty' only (possibly with 'empty' -> 'missing'), if there is no 'Content-Length or Transfer-Encoding header' for a response with an HTTP error code needing a message body?

[Lutz]

Fixed in: http://www.newlisp.org/downloads/develo ... nprogress/

Code: Select all

> (get-url "http://localhost:8080/empty.txt" "debug")
GET /empty.txt HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close

HTTP/1.0 200 OK

""
>


[hartrock]

Hello Lutz,

first thank you for the very fast reaction!

Checking last (just downloaded) devel version gives problems:

Code: Select all

sr@mad:/tmp/newlisp-10.6.4$ make check
./newlisp qa-dot

Testing built-in functions ...
make: *** [check] Segmentation fault


But:

Code: Select all

sr@mad:/tmp/newlisp-10.6.4$ ./newlisp qa-dot

Testing built-in functions ...
upper-case of UTF8 characters not available - not critical -
lower-case of UTF8 characters not available - not critical -
                             
Testing contexts as objects and scoping rules ...

total time: 2730

>>>>> ALL FUNCTIONS FINISHED SUCCESSFUL: ./newlisp

sr@mad:/tmp/newlisp-10.6.4$ make testall
make checkall | grep '>>>'
Segmentation fault
>>>>> Dictionary API tested SUCCESSFUL
>>>>> XML API tested SUCCESSFUL
>>>>> XML callback tested SUCCESSFUL
>>>>> JSON translation tested SUCESSFUL
>>>>> Signal testing SUCCESSFUL
>>>>> Network eval and network file functions IPv4 SUCCESSFUL
>>>>> Network eval and network file functions IPv6 SUCCESSFUL
>>>>> The Cilk API tested SUCCESSFUL
>>>>> Reference testing SUCCESSFUL
>>>>> Time per simple message: 8 micro seconds
>>>>> Time per round trip : 70 micros seconds
>>>>> Time per proxy trip: 20 micro seconds
>>>>> Message API tested SUCCESSFUL
>>>>> Memory allocation/deallocation SUCCESSFUL
>>>>> Exception testing SUCCESSFUL
>>>>> Floating point tests SUCCESSFUL
>>>>> FOOP nested 'self' tested SUCCCESSFUL
>>>>> FOOP symbol protection SUCCESSFUL
>>>>> UNIX local domain sockets SUCCESSFUL
>>>>> In-place modification passed SUCCESSFUL
>>>>> 0.005 ms per write->read pipe/fork (0.0356 ms Mac OSX, 1.83 GHz Core 2 Duo)
>>>>> libffi API testing SUCCESSFUL
>>>>> callback API 1234567890 (simple callback API) SUCCESSFUL
>>>>> callback API 1234567890 12345.6789 (extended callback API) SUCCESSFUL
>>>>> struct tested SUCCESSFUL
>>>>> struct ptr tested SUCCESSFUL
>>>>> testing big ints arithmetik ...
>>>>> abs bigint float gcd length zero? + - * / % ++ -- big ints tested SUCCESSFUL
>>>>> parsing big integers SUCCESSFUL
>>>>> Benchmarking all non I/O primitives ... (may take a while)
>>>>> total time: 1503.266
>>>>> Performance ratio: 0.62 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)


Digging deeper this difference in behavior is suspect:

Code: Select all

sr@mad:/tmp/newlisp-10.6.4$ make check | more
./newlisp qa-dot

Testing built-in functions ...
  -> >>         
  -> array               
  -> begin               
  -> clean               
  -> cpymt                 
  -> def-                 
  -> div                   
  -> error             
  -> file?               
  -> gammai           
  -> inc                 
  -> lambda?train         
  -> macro-case         
  -> net-close           
  -> net-receive           
  -> now                       
  -> pop-               
  -> prompt-event         
  -> read-file           
  -> replace               
  -> series             
make: *** [check] Segmentation fault
  -> share               
sr@mad:/tmp/newlisp-10.6.4$ ./newlisp qa-dot | more

Testing built-in functions ...
  -> >>         
  -> array               
  -> begin               
  -> clean               
  -> cpymt                 
  -> def-                 
  -> div                   
  -> error             
  -> file?               
  -> gammai           
  -> inc                 
  -> lambda?train         
  -> macro-case         
  -> net-close           
  -> net-receive           
  -> now                       
  -> pop-               
  -> prompt-event         
  -> read-file           
  -> replace               
  -> series             
  -> sort                 
  -> sys-error           
  -> transpose   ght         
upper-case of UTF8 characters not available - not critical -
lower-case of UTF8 characters not available - not critical -
  -> xml-parse             
                             
Testing contexts as objects and scoping rules ...

total time: 2753

>>>>> ALL FUNCTIONS FINISHED SUCCESSFUL: ./newlisp



First checks 'share', second not?

OS:

Code: Select all

sr@mad:/tmp/newlisp-10.6.4$ uname -a
Linux mad 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u2 x86_64 GNU/Linux


[Lutz]

Fixed here: http://www.newlisp.org/downloads/develo ... nprogress/

[hartrock]

Thanks: OK now!

Another thing (kind of feature request):
What about

(http-error) for more fine-granular checking of http responses?

Currently I have client code like (snippet):

Code: Select all

        (setq res (get-url (append "http://localhost/ViPLab/" r))
              net_err (net-error))
        (if (not net_err)
            (if (not (find "^ERR: server code 404:" res 0))
                (print res)
                (begin
                  (logg:err (0 (find "\n" res) res))
                  (exit 1)))
            (begin
              (logg:err net_err)
              (exit 1)))


There may be a need to extend this further for other error messages (formatting stable?)...

It would be nice to have a unique and simple access to last HTTP status/error code (may be nil for no or (syntactically) invalid server response).
It already will be parsed somewhere...

Then simple error checking could be done by (htt-status is better than http-error)

Code: Select all

(>= (http-status) 400)


or so (besides checking for net-error).

(get-url) "list" mode does not help for a unique access:
there is no HTTP status (error or not) in headers part (which I would expect there, because it is a header), but only some error messages (covering which HTTP status codes?) prepended to content from server; and if there is no error (which HTTP status codes?), there is no access to HTTP status code at all (there is more no-error status codes than just '200 OK').

[Lutz]

The main reason for having server mode in newLISP is for distributed processing with ‘net-eval’. HTTP mode was added to make simple browser configuration pages for servers possible, e.g. for configuring master-nodes in distributed system and configuring embedded systems like modems. It was never meant to be full a featured web server.

[hartrock]

Let me explain my current usecase a bit (there is no use of newLISP -http mode).

Usecase (being topic in this thread) is HTTP-server testing by newLISP scripts, being the clients of an HTTP server.
Therefore I wanted to get rid of calling curl commands by newLISP scripts (after getting rid of curl commands as part of bash scripts). Why using curl, if there already is similar functionality built in newLISP?

So I've started using (get|post|delete|put)-url funcs, which looked promising for this purpose.
But for fully testing server behavior, it makes sense to have access to whole server response - headers, content (which mostly is part of a server reply) and HTTP status code in a unique way.

Because there already is

• some parsing of server responses for creating HTTP net-errors,
• "devel" mode for *-url requests, giving almost all headers;

at first sight less effort seems to be needed for a great improvement of server testing capabilities.
But I may be wrong here, because there surely are some complications (from the manual):

get-url handles redirection if it detects a Location: spec in the received header and automatically does a second request. get-url also understands the Transfer-Encoding: chunked format and will unpack data into an unchunked format.

And if someone wanted to test e.g. redirection behavior of some server, then there may be further feature requests... (but not from me ;-) )

I'm assuming, that implementing such functionality as a module - funcs possibly named http-(get|post|delete|put|head)-url - would double much stuff already there as part of built in functions (and reuse instead of doubling is a good thing).

Notes to tested server
Incidentally to be tested server is a newLISP one, currently reached via Apache by mod_lisp protocol (not so far from HTTP); this newLISP server is listening for Apache requests.
Request:
- client -> Apache -> newLISP-Server
Reply:
- client <- Apache <- newLISP-Server
. For handling Apache requests after mod_lisp protocol, net-listen, net-accept, net-receive, etc. will be used.
Later there may be a newLISP-Server directly serving HTTP requests, but surely not by using newLISP -http mode.

Note:
Had tried newLISP -http mode first to get some web server, but - as you already said - it'is limited, because

It was never meant to be full a featured web server.

[Lutz]

In 'get/put/post-url' the new string option "raw" works like the "list" option but returns the server status/error code as the the last member in the returned list and also suppresses 'Location' redirection.

http://www.newlisp.org/downloads/develo ... nprogress/

[hartrock]

Thanks: this is good for my usecase (and probably some of others).

Some feedback:

• Inconsistency:
  Using additional "debug" flag, "list debug" differs from "raw debug", which does not show sent headers:
  

  Code: Select all

  > (get-url "http://localhost/doesNotExist.txt" "list debug")
GET /doesNotExist.txt HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close

HTTP/1.1 404 Not Found

("Date: Fri, 17 Jul 2015 11:17:21 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nVary: Accept-Encoding\r\nContent-Length: 289\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /doesNotExist.txt was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>\n</body></html>\n"
 404)
> (get-url "http://localhost/doesNotExist.txt" "raw debug")
("Date: Fri, 17 Jul 2015 11:17:30 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nVary: Accept-Encoding\r\nContent-Length: 289\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /doesNotExist.txt was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>\n</body></html>\n"
 404)


  
  I would expect to see sent headers for "raw debug", too.
• Docu:
  Difference in semantics between "raw" and "list" seems to be, to avoid redirection in "raw" case (not tested); because both return the HTTP status code as third element (which is good and should stay IMO):
  

  Code: Select all

  > (get-url "http://localhost/doesNotExist.txt" "list")
("Date: Fri, 17 Jul 2015 11:18:07 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nVary: Accept-Encoding\r\nContent-Length: 289\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /doesNotExist.txt was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>\n</body></html>\n"
 404)
> (get-url "http://localhost/doesNotExist.txt" "raw")
("Date: Fri, 17 Jul 2015 11:18:11 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nVary: Accept-Encoding\r\nContent-Length: 289\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /doesNotExist.txt was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>\n</body></html>\n"
 404)


[Lutz]

Thanks for the thorough analysis. Changed option handling again:

The "raw" option only suppresses the location redirection and can be used alone or combined with other options. The status code is always included as the last list member when the "list" option is specified.

http://www.newlisp.org/downloads/develo ... nprogress/

try:

Code: Select all

(get-url "http://newlisp.org")
(get-url "http://newlisp.org" "raw")
(get-url "http://newlisp.org" "list raw")
(get-url "http://newlisp.org" "list raw debug")


[hartrock]

Thanks for the upgrade.

Unfortunately there is some problem:

Code: Select all

> ; "OK with "list":
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
("Date: Fri, 17 Jul 2015 18:37:10 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> ; -> OK
> ;
> ; but with "list debug":
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Fri, 17 Jul 2015 18:38:13 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> ; -> failure: see the string artefact 'ian)'
> ;
> ; it can be repaired by:
> (print (0 (t 1)))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"


This is very strange. Possibly the 'repair' by copying the string gives you an idea, where the problem may be...

Digging deeper:

Code: Select all

> (set 'r1 (t 1))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (print r1)
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'r2 (0 r))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (print r2)
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (list (length r1) (length r2))
(63 63)
> (map (lambda (ix) (= (r1 ix) (r2 ix))) (sequence 0 (-- (length r1))))
(true true true true true true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true true true true true true)


At interpreter level strings seem to be OK...

[Lutz]

If this happened on Windows or OS2, TRU64, Solaris or AIX, it may have been fixed:

http://www.newlisp.org/downloads/develo ... nprogress/

There was an uninitialized buffer in a custom version of vasprintf() -> my_vasprintf() used by above OSs.

Ps: doing some more changes affecting all OSs, the way varargs is used - will post those Sunday morning PST.

[hartrock]

If this happened on Windows or OS2, TRU64, Solaris or AIX, it may have been fixed:

http://www.newlisp.org/downloads/develo ... nprogress/

There was an uninitialized buffer in a custom version of vasprintf() -> my_vasprintf() used by above OSs.

No, error has been on Debian Linux.

Ps: doing some more changes affecting all OSs, the way varargs is used - will post those Sunday morning PST.

This sounds like a bigger change with many chances for errors...

When there is a newer devel version I'll test again on Monday or other working days and give feedback again (I don't have access to corresponding system here).

[Lutz]

Another update avoiding empty argptr in variadic nl-web.c:sendf() :

http://www.newlisp.org/downloads/develo ... nprogress/

[hartrock]

Testing with

Code: Select all

newlisp-10.6.4.tgz      2015-07-19 23:31  1.6M

bug stays (first call OK, thereafter bug):

Code: Select all

newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
("Date: Mon, 20 Jul 2015 13:51:33 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> ; -> OK
> ;
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 13:52:17 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> ; -> bug triggered
> ;


Another observation: if at first provoking the bug with "list debug", then it stays even with arg "list" not provoking it in former test, but - update - only once:

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 14:06:13 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
("Date: Mon, 20 Jul 2015 14:06:34 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list"))
("Date: Mon, 20 Jul 2015 14:18:50 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
>


Anything else I could test?

BTW: it would be helpful, if

Code: Select all

newlisp -v

(or -v -v) would contain a built timestamp (to be sure, that you are really using a just built binary (though just building/installing does not ensure, that you have built from the correct source)).

[Lutz]

Have tried your experiment on OSX, Windows 7, UBUNTU and FreeBSD with your strings and others, but cannot repeat that spurious string occuring. Here some other small change:

http://www.newlisp.org/downloads/develo ... nprogress/

[hartrock]

Bug stays with

Code: Select all

newlisp-10.6.4.tgz      2015-07-20 15:25  1.6M

:

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 16:13:26 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 16:13:48 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
>


Additional observation: with triggering bug and printing in one line, at very first time only there is no bug to see, but then it appears and stays:

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug")) (print (t 1))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 16:09:54 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug")) (print (t 1))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 16:10:06 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> ; repeating the same from here bug stays.


It's difficult for me to trigger this bug under other circumstances; one example: if there is a 404er directly from Apache (without newLISP server behind), there is no bug to see:

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/foo" "dummy content" "text/plain" "list debug"))
POST /foo HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 16:36:38 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nVary: Accept-Encoding\r\nContent-Length: 276\r\nConnection: close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /foo was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>\n</body></html>\n"
 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /foo was not found on this server.</p>
<hr>
<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>
</body></html>
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /foo was not found on this server.</p>\n<hr>\n<address>Apache/2.2.22 (Debian) Server at localhost Port 80</address>\n</body></html>\n"
>


Don't know, how to come nearer to the problem this way.

Is there anything, which I could check in direction of 'digging deeper'?
Like here:

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 17:48:02 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (set 'r (t 1))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'rc (0 r))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (list (length r) (length rc))
(63 63)
> (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
(true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true)
> (= r rc)
nil
> (println r)
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

i
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (println rc)
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.


"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> ;
> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Mon, 20 Jul 2015 17:52:31 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (println (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
>


That both length and chars of strings are '=', but not their comparison with '=' is very strange.
[update] utf8len is equal, too:

Code: Select all

> (list (utf8len r) (utf8len rc))
(63 63)


Compiler:

Code: Select all

sr@mad:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.7.2 (Debian 4.7.2-5)


[hartrock]

Bug stays with (info taken from http://www.newlisp.org/downloads/develo ... nprogress/)

Code: Select all

newlisp-10.6.4.tgz      2015-07-20 23:50  1.6M

(dump output of differing strings added):

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Tue, 21 Jul 2015 09:22:58 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'r (t 1))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'rc (0 r))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
(63 63)
(63 63)
> (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
(true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true)
> (dump r)
(139910737302512 260 139910737289232 64 19955600)
> (dump rc)
(139910737302448 260 139910737289232 64 19994480)
>


Is it normal, that both have same 'cell->next: linked list ptr'?

[Lutz]

That is the address of the original nil cell in the system and is normal.

May be it has to do with UTF8 handling, although all my tests are fine without it. Can you compile without UTF8 and try again?

[hartrock]

Found interesting difference in strings (see at bottom):

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Tue, 21 Jul 2015 14:49:31 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'r (t 1))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'rc (0 r))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
(63 63)
(63 63)
> (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
(true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true)
> (dump (t 1)) (dump r) (dump rc)
(140457260381456 260 140457260381520 64 20146112)
(140457260381168 260 140457260367888 64 20107152)
(140457260381104 260 140457260367888 64 20146032)
> (address (t 1)) (address r) (address rc)
20146112
20107152
20146032
> (set 's "____________________")
"____________________"
> (cpymem (+ (address r) 50) (address s) 20) s
20
" Not Found.\n\nian)\r\n\000"
> (cpymem (+ (address rc) 50) (address s) 20) s
20
" Not Found.\n\n\000\000\000\000\000\000\000"
>


So '\0' comes to late in (t 1) and r, but not in rc.
Interpretation:

• String cell->contents have been copied in both copy cases (set 'r (t 1)) and ((set 'rc (0 r)) (see dump).
• Copying string cell->contents by (0 r) honors string length in cell and ensures ending its content with a '\0'; but (t 1) does only a copy until a '\0' byte in string content of source, without looking for string length.

Hope that helps.

[hartrock]

May be it has to do with UTF8 handling, although all my tests are fine without it. Can you compile without UTF8 and try again?

Same result:

Code: Select all

sr@mad:/tmp/newlisp-10.6.4$ ./newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Tue, 21 Jul 2015 15:36:29 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

ian)
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'r (t 1))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'rc (0 r))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
(63 63)

ERR: invalid function in function list : (utf8len r)
> (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
(true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true)
> (dump (t 1)) (dump r) (dump rc)
(140371457590416 260 140371457590480 64 29344512)
(140371457590128 260 140371457576976 64 29343904)
(140371457590064 260 140371457576976 64 29344432)
> (address (t 1)) (address r) (address rc)
29344512
29343904
29344432
> (set 's "____________________")
"____________________"
> (cpymem (+ (address r) 50) (address s) 20) s
20
" Not Found.\n\nian)\r\n\000"
> (cpymem (+ (address rc) 50) (address s) 20) s
20
" Not Found.\n\n\000\000\000\000\000\000\000"
>

[Lutz]

Many thanks! I think it is solved now. The cpymem test clearly showed the missing zero termination of the response string in the cell. This happened for content appended to an error message.

newLISP also stores the size of a memory buffer independent of string zero-termination. Because of this, the error doesn't always show up.

http://www.newlisp.org/downloads/develo ... nprogress/ from 2015-07-21 20:45

Ps: May be newLISP shouldn't prepend and error message to an exsisting server error page at all. Perhaps the error message should only be generated when no page comes back from server after the header.

[hartrock]

Many thanks! I think it is solved now. The cpymem test clearly showed the missing zero termination of the response string in the cell. This happened for content appended to an error message.

newLISP also stores the size of a memory buffer independent of string zero-termination. Because of this, the error doesn't always show up.

http://www.newlisp.org/downloads/develo ... nprogress/ from 2015-07-21 20:45

Ps: May be newLISP shouldn't prepend and error message to an exsisting server error page at all. Perhaps the error message should only be generated when no page comes back from server after the header.

Good news: bug has gone!

Code: Select all

sr@mad:~/newLISP/BoF$ newlisp
newLISP v.10.6.4 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (set 't (post-url "http://localhost/ViPLab" "dummy content" "text/plain" "list debug"))
POST /ViPLab HTTP/1.1
Host: localhost
User-Agent: newLISP v10604
Connection: close
Content-type: text/plain
Content-length: 13

dummy contentHTTP/1.1 404 Not Found

("Date: Wed, 22 Jul 2015 11:50:22 GMT\r\nServer: Apache/2.2.22 (Debian)\r\nContent-Length: 17\r\nConnection: close\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n"
 "ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n" 404)
> (print (t 1))
ERR: server code 404: HTTP/1.1 404 Not Found
404: Not Found.

"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'r (t 1))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (set 'rc (0 r))
"ERR: server code 404: HTTP/1.1 404 Not Found\r\n404: Not Found.\n\n"
> (list (length r) (length rc)) (list (utf8len r) (utf8len rc))
(63 63)
(63 63)
> (map (lambda (ix) (= (r ix) (rc ix))) (sequence 0 (-- (length r))))
(true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true true
 true true true true true true true true true true true true true true true)
> (dump (t 1)) (dump r) (dump rc)
(140043466192144 260 140043466192208 64 13359008)
(140043466191856 260 140043466178576 64 13320048)
(140043466191792 260 140043466178576 64 13358928)
> (address (t 1)) (address r) (address rc)
13359008
13320048
13358928
> (set 's "____________________")
"____________________"
> (cpymem (+ (address r) 50) (address s) 20) s
20
" Not Found.\n\n\000an)\r\nC"
> (cpymem (+ (address rc) 50) (address s) 20) s
20
" Not Found.\n\n\000\000\000\000\000\000\000"
>


[hartrock]

Ps: May be newLISP shouldn't prepend and error message to an exsisting server error page at all. Perhaps the error message should only be generated when no page comes back from server after the header.

Short feedback:

• Sounds like an improvement, because typical server behavior is to give some info about the error (which can be more detailed than just having the HTTP status code/message).
• A further improvement for clients handling the HTTP status computationally could be:
  
  • return nil for no content, or "" for empty one (content-length 0);
  • introduce (http-status) for getting status code or nil, if there is none (because of some other error); this should work in all modes ("", "list", "list raw").

Longer feedback

There are different levels, where an error can arise:

1. low-level: (net-error).
2. HTTP request level: check for correct HTTP message format.
   Second: similar to "3. Second:".
3. HTTP response level: check for correct HTTP message format first.
   Second: there may be a correct HTTP message format with some HTTP status code, but nevertheless there may be incorrect HTTP behavior; e.g.
   
   • wrong HTTP status code;
     
   • a message body where none is allowed, or no message body where it is needed.
4. HTTP status code: if all is OK until here, HTTP status code may indicate success or errors higher level.
5. HTTP protocol level: correct server behavior comprising multiple requests? E.g. a DELETE followed by a GET of the same resource should give a 410 or 404 status.
6. server high level: functional behavior, internal server errors (returning some 5xx'er status code), etc..

Some of net-errors are lower level, some are HTTP related; from the manual:
1 - 18: lower level errors
19 HTTP bad formed URL
20 HTTP file operation failed
21 HTTP transfer failed
22 HTTP invalid response from server
23 HTTP no response from server
24 HTTP document empty
25 HTTP error in header
26 HTTP error in chunked format

Some questions and comments to these error codes:
21 HTTP transfer failed:
Which error cases are covered here?
22 HTTP invalid response from server:
Which error cases are covered here? All at 'HTTP response level' above? Or more at higher levels above, too?
23 HTTP no response from server:
What's the difference to '18 Operation timed out'?
24 HTTP document empty:
An empty document in sense of empty content is not an error, but missing empty content (without 'Content-Length or Transfer-Encoding header field' with empty message body).
25 HTTP error in header:
Probably not in client request, but server response. If so: how to get errors in client request headers?
26 HTTP error in chunked format:
Same question as in 25.

Ideas (to be made more mature) for improvement:

1. • Reduce HTTP errors returned by (net-error) to one - e.g. "19 HTTP error";
   • introduce (http-error) for more fine-granular HTTP errror handling: e.g.
     
     • error request header URL (related to specific header entry)
     • ... (other specific header entries)
     • error request headers (related to all headers)
     • error request format (related to whole message (headers and body))
     • error response header xxx (specific header)
     • error response headers
     • error response format chunked (specific)
     • error response format (general)
     • error protocol (request and response seen together one or more times).
     • ...
2. More fine-granular HTTP errror handling from a. could be put flat into (net-error) errors as well: but this does not reflect the different levels the errors are arising from.

Problems seen yet:

• A more clear separation of different protocol levels for finer error control in some protocol stack stands against the goal of reducing error handling complexity for typical use cases.
• Going this or a similar route makes work (and time is limited for all of us). Just writing this post...

[Lutz]

Thanks for the input.