Newlisp-Apache-CGI Issues Again

Machine-specific discussion
Unix, Linux, OS X, OS/2, Windows, ..?
dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Newlisp-Apache-CGI Issues Again

Post by dukester »

PS C:\bin\newlisp> newlisp -v
newLISP v.10.7.1 64-bit on Windows IPv4/6 UTF-8 libffi.

Running Apache2.4 on a Win10 box.

Apache is choking - once again - on the header sent by newlisp:
Content-type' contains invalid characters, aborting request
Lutz fixed a similar issue back in 2015, but it seems to have re-surfaced!

This is the code used then and now:

Code: Select all

#!C:\bin\newlisp\newlisp.exe
(print "Content-type: text/html\n\n")

(print "<html>")
(print "<h4>CGI by newLISP v" (sys-info -2)" on " ostype "</h4>")
(dolist (e (env)) (println (e 0) " : " (e 1) "</br>"))
(println "</html>")
(exit)
Anybody know how to fix this issue?
duke

rickyboy
Posts: 607
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Post by rickyboy »

Looks suspiciously like the ole windoze versus unix line ending issue/difference. I’d check that before moving on to anything else.

For instance, what about trying this instead? (Sorry, “I don’t do windows.” :)

Code: Select all

(println "Content-type: text/html")
(println "")
Or something like that? Good luck!
(λx. x x) (λx. x x)

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

rickyboy wrote:Looks suspiciously like the ole windoze versus unix line ending issue/difference. I’d check that before moving on to anything else.
Nope! It's a newlisp issue - just like back in 2015.

I tried your suggestion, even though I knew it wouldn't work. And it didn't. Lutz will have to get involved I'm sure. Thanks anyway!
duke

ralph.ronnquist
Posts: 228
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: Newlisp-Apache-CGI Issues Again

Post by ralph.ronnquist »

You might want to try the initial line

Code: Select all

Status: 200 OK

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

ralph.ronnquist wrote:You might want to try the initial line

Code: Select all

Status: 200 OK
Don't know what you mean!
duke

ralph.ronnquist
Posts: 228
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: Newlisp-Apache-CGI Issues Again

Post by ralph.ronnquist »

afair, a CGI response that includes headers should have a very first line being the HTTP response status code, then the header lines, then an empty line, and then the payload.

You may omit the headers part and start with the empty line, and then the CGI runner would insert a standard header component (with a default status line).

But not all CGI runners insert a status line before headers when headers are given. For example, newlisp does insert a default status line and default headers (in its -http mode CGI runner), but possibly apache2 doesn't.

Though, I do this irregularly enough that I typically have to refer to the CGI spec's to remind myself about the nitty-gritty, and apache2 is (like most) a moving target anyhow.

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

@ralph.ronnquist - I'm sure that you are absolutely correct. However, I've never had to do what you are suggesting. Most of my experience has been on Apache using from CL to Perl to crank out CGI stuff. Newlisp is the only interpreter that has ever given me fits with these type of issues - given that I didn't trash that very important first line of code.

Lutx fixed this issue once. He may not want to deal with it again. I've got the very same script running just fine using CLISP2.49. I don't use -http unless I want to serve an Intranet.

thx for getting back to me.
duke

rickyboy
Posts: 607
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Post by rickyboy »

What ralph said. Apache 2.4 got "strict" with the header validations to mitigate vulnerabilities when used as an http proxy, e.g. this one in 2016 (interesting read too):

https://httpd.apache.org/security/vulne ... -2016-8743

Notice the switch where you can turn that off (although not recommended):

http://httpd.apache.org/docs/2.4/mod/co ... coloptions

Could that have been the reason that that code worked recently and now it doesn't? (Switch got changed?)

I think keeping that setting to the default strict mode and going down the path that ralph suggested for your troubleshooting sounds good to me -- at least in the meantime before you hear anything from Lutz. Again, good luck.

(If I had the same stack, I could try to recreate the problem; however. along with not being a Windows user, I'm also not an Apache user. :)
(λx. x x) (λx. x x)

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

@rickyboy All good stuff! But if Apache was being a bad-ass, it would be doing so with my Perl, Python and Ruby scripts! Even old CLISP scripts run well. Get where I'm going with this? ;-)

Thanks for the heads-up regarding Apache security stuff!
duke

xytroxon
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm
Contact:

Re: Newlisp-Apache-CGI Issues Again

Post by xytroxon »

I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.

Change the line:

Code: Select all

#!C:\bin\newlisp\newlisp.exe
To:

Code: Select all

#!C:\bin\newlisp\newlisp.exe -c
-- xytroxon
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

[quote="xytroxon"]I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.

Change the line:

Code: Select all

#!C:\bin\newlisp\newlisp.exe
To:

Code: Select all

#!C:\bin\newlisp\newlisp.exe -c
Thx! I'll give that a shot!!
duke

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

Nope! No cigar! LOL
duke

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Newlisp-Apache-CGI Issues Again

Post by Lutz »

Try:

Code: Select all

(print "Content-Type: text/html\r\n\r\n")

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

@lutz Thx, but that does not work either. Sorry!
duke

ralph.ronnquist
Posts: 228
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: Newlisp-Apache-CGI Issues Again

Post by ralph.ronnquist »

Have you verified the byte sequence from the script, eg by running it manually and pipe its output into a file, which you the inspect with a hex code viewer? Or even perhaps like the following:

Code: Select all

% thescript | C:\bin\newlisp\newlisp.exe -e '(read 0 b 10000)(map char (explode b))'
Note, I don't really use windows myself, so I'm guessing about piping the output from one command (thescript) to another (the unpacking into character codes). In any case, the resulting character codes list should then begin

Code: Select all

(67 111 110 116 101 110 116 45 116 121 112 101 ...

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

@ralph.ronnquist
You bet that I have an Hex editor!! I don't see any garbage - which puzzles me. I've included a screen capture of the output. Now I'm wondering what Apache is choking on!! Pain in the Pa-toot!! LOL
Attachments
outputData1.png
outputData1.png (28.68 KiB) Viewed 14014 times
duke

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Newlisp-Apache-CGI Issues Again

Post by Lutz »

Check how NEWLISPDIR and PROGRAMFILES are defined.

See here for an explanation of these directories: http://www.newlisp.org/downloads/newlis ... ml#startup

You could use the fowlowing script to check:
(println (env "PROGRAMFILES")) ; only defined on Windows
(println (env "NEWLISPDIR"))
(exit)

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

Both are set. I've installed newlisp in c:\bin\newlisp to make sure that I did not have future issues with spaces in directory names like "Program Files" etc. So NEWLISPDIR points to c:\bin\newlisp.

PROGRAMFILES is set to "Program Files". I changed it to c:\bin\newlisp.

The change did not fix my problem. BTW what did you do in 2015 to fix this very same issue? You modified something in newlisp.exe and asked me to try your modifications. One of them worked!!! :-)
duke

rickyboy
Posts: 607
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Post by rickyboy »

duke,

Hex dump seems ok to me. I'm assuming you did something like:

Code: Select all

C:\> newlisp-cgi-script > newlisp-cgi.out
Then, looked at newlisp-cgi.out in the hex editor. Maybe try also something like:

Code: Select all

C:\> perl-cgi-script > perl-cgi.out
and maybe the same with the Ruby, CLISP scripts, etc. (i.e., all the CGI scripts that Apache doesn't choke on), and see if you can spot any difference between each output from those and newlisp-cgi.out.
(λx. x x) (λx. x x)

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Newlisp-Apache-CGI Issues Again

Post by Lutz »

Try the following:

Create a subdirectory "newlisp" in "Program Files" and put the newlisp.exe executable in there. If there are any modules you are going to use, create a "modules" directory inside the "newlisp" directory.

Code: Select all

(env "PROGRAMFILES") => C:\Program Files (x86)
(env "NEWLISPDIR") => C:\Program Files (x86)\newlisp
Path of newlisp.exe is now: C:\Program Files (x86)\newlisp\newlisp.exe

So both environment variables point to directories and the path in NEWLISPDIR must be in the executable path.

May be Windows finds the newlisp executable when using the command line, but not when called from Apache.

The original fix for Apache in 2015 was putting defaults (the above) for both environment variables.

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

@Lutz Thanks for all the trouble you went through!! Unfortunately, after doing all your recommendations, nothing changed! No Cigars!! I still get the same response from Apache:
[Fri Mar 29 09:35:53.040187 2019] [http:error] [pid 10048:tid 1076] [client 127.0.0.1:51447] AH02429: Response header name 'Environment variable TMP not set, assuming /tmp .Content-Type' contains invalid characters, aborting request
I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?
duke

rickyboy
Posts: 607
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Post by rickyboy »

dukester wrote:I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?
That’s not what your hex dump indicates.

It may be due to a configuration/interface issue — I believe that’s the trail Lutz was going down and a good one to pursue.

It would be nice to see what text stream Apache is seeing. It may be something very unlike what we are imagining in our heads. Any way to do that? IOW, does Apache have a better way to diagnose these problems (like a debug level) because that exception message that you get tells us hardly anything at all and it may even be “lying” to us.

Also, if you don’t know already, you may want to research into how Apache is calling out to / running your script. There may be some devil in the details there. If they didn’t doc it well, you might have to go code spelunking. Ugh.
(λx. x x) (λx. x x)

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

@rickyboy You know - I'm thinking that Lutz has already wasted enough time looking into this issue. If there were hundreds of folks using newlisp on Apache, it might then be a worthwhile investment of time. I'm sure that the folks at Apache would pretty much feel the same way! How many people are they aware of that use Apache to run newlisp scripts. I'm sure they couldn't be bothered.

How's the song go? "You got to know when to hold 'em, know when to fold 'em, Know when to walk away and know when to run. " Maybe that time is come. I'll pick at it now and then, but let's not waste anymore time. I thought the solution might be simple, and already solved. But there I went thinking again! LOL Thanks anyway! Everybody ....
duke

rickyboy
Posts: 607
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Post by rickyboy »

duke, I wasn't saying for Lutz to look at it, but for *you* to look at it. lol!

No worries, I get it. I've been there many times. We do indeed have to make that determination: whether to spelunk or not. :) Sometimes, it's not worth it, especially when we have time constraints. I hope you get some time here and there to mess around with it ... and succeed!

All the best! rickyboy
(λx. x x) (λx. x x)

dukester
Posts: 115
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Post by dukester »

Yeah! I realized that you meant me! I was just trying to be civil and say "No $%%^^* way!!" LOL And Lutz does have better things to do! Thanks - I will fiddle with it as and when the mood hits me. Good luck yourself!
duke

Locked