Newlisp-Apache-CGI Issues Again

Machine-specific discussion
Unix, Linux, OS X, OS/2, Windows, ..?

Newlisp-Apache-CGI Issues Again

Postby dukester » Tue Mar 26, 2019 1:28 pm

PS C:\bin\newlisp> newlisp -v
newLISP v.10.7.1 64-bit on Windows IPv4/6 UTF-8 libffi.

Running Apache2.4 on a Win10 box.

Apache is choking - once again - on the header sent by newlisp:

Content-type' contains invalid characters, aborting request


Lutz fixed a similar issue back in 2015, but it seems to have re-surfaced!

This is the code used then and now:
Code: Select all
#!C:\bin\newlisp\newlisp.exe
(print "Content-type: text/html\n\n")

(print "<html>")
(print "<h4>CGI by newLISP v" (sys-info -2)" on " ostype "</h4>")
(dolist (e (env)) (println (e 0) " : " (e 1) "</br>"))
(println "</html>")
(exit)


Anybody know how to fix this issue?
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby rickyboy » Tue Mar 26, 2019 6:56 pm

Looks suspiciously like the ole windoze versus unix line ending issue/difference. I’d check that before moving on to anything else.

For instance, what about trying this instead? (Sorry, “I don’t do windows.” :)

Code: Select all
(println "Content-type: text/html")
(println "")

Or something like that? Good luck!
(λx. x x) (λx. x x)
rickyboy
 
Posts: 568
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Tue Mar 26, 2019 8:47 pm

rickyboy wrote:Looks suspiciously like the ole windoze versus unix line ending issue/difference. I’d check that before moving on to anything else.

Nope! It's a newlisp issue - just like back in 2015.

I tried your suggestion, even though I knew it wouldn't work. And it didn't. Lutz will have to get involved I'm sure. Thanks anyway!
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby ralph.ronnquist » Tue Mar 26, 2019 9:40 pm

You might want to try the initial line
Code: Select all
Status: 200 OK
ralph.ronnquist
 
Posts: 188
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Tue Mar 26, 2019 9:55 pm

ralph.ronnquist wrote:You might want to try the initial line
Code: Select all
Status: 200 OK

Don't know what you mean!
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby ralph.ronnquist » Tue Mar 26, 2019 10:46 pm

afair, a CGI response that includes headers should have a very first line being the HTTP response status code, then the header lines, then an empty line, and then the payload.

You may omit the headers part and start with the empty line, and then the CGI runner would insert a standard header component (with a default status line).

But not all CGI runners insert a status line before headers when headers are given. For example, newlisp does insert a default status line and default headers (in its -http mode CGI runner), but possibly apache2 doesn't.

Though, I do this irregularly enough that I typically have to refer to the CGI spec's to remind myself about the nitty-gritty, and apache2 is (like most) a moving target anyhow.
ralph.ronnquist
 
Posts: 188
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Tue Mar 26, 2019 11:02 pm

@ralph.ronnquist - I'm sure that you are absolutely correct. However, I've never had to do what you are suggesting. Most of my experience has been on Apache using from CL to Perl to crank out CGI stuff. Newlisp is the only interpreter that has ever given me fits with these type of issues - given that I didn't trash that very important first line of code.

Lutx fixed this issue once. He may not want to deal with it again. I've got the very same script running just fine using CLISP2.49. I don't use -http unless I want to serve an Intranet.

thx for getting back to me.
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby rickyboy » Tue Mar 26, 2019 11:10 pm

What ralph said. Apache 2.4 got "strict" with the header validations to mitigate vulnerabilities when used as an http proxy, e.g. this one in 2016 (interesting read too):

https://httpd.apache.org/security/vulne ... -2016-8743

Notice the switch where you can turn that off (although not recommended):

http://httpd.apache.org/docs/2.4/mod/co ... coloptions

Could that have been the reason that that code worked recently and now it doesn't? (Switch got changed?)

I think keeping that setting to the default strict mode and going down the path that ralph suggested for your troubleshooting sounds good to me -- at least in the meantime before you hear anything from Lutz. Again, good luck.

(If I had the same stack, I could try to recreate the problem; however. along with not being a Windows user, I'm also not an Apache user. :)
(λx. x x) (λx. x x)
rickyboy
 
Posts: 568
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Tue Mar 26, 2019 11:41 pm

@rickyboy All good stuff! But if Apache was being a bad-ass, it would be doing so with my Perl, Python and Ruby scripts! Even old CLISP scripts run well. Get where I'm going with this? ;-)

Thanks for the heads-up regarding Apache security stuff!
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby xytroxon » Wed Mar 27, 2019 2:42 am

I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.

Change the line:
Code: Select all
#!C:\bin\newlisp\newlisp.exe


To:
Code: Select all
#!C:\bin\newlisp\newlisp.exe -c


-- xytroxon
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976
xytroxon
 
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Wed Mar 27, 2019 3:04 am

[quote="xytroxon"]I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.

Change the line:
Code: Select all
#!C:\bin\newlisp\newlisp.exe


To:
Code: Select all
#!C:\bin\newlisp\newlisp.exe -c


Thx! I'll give that a shot!!
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Wed Mar 27, 2019 3:12 am

Nope! No cigar! LOL
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby Lutz » Wed Mar 27, 2019 3:21 am

Try:

Code: Select all
(print "Content-Type: text/html\r\n\r\n")
Lutz
 
Posts: 5269
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Wed Mar 27, 2019 10:35 am

@lutz Thx, but that does not work either. Sorry!
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby ralph.ronnquist » Wed Mar 27, 2019 12:42 pm

Have you verified the byte sequence from the script, eg by running it manually and pipe its output into a file, which you the inspect with a hex code viewer? Or even perhaps like the following:
Code: Select all
% thescript | C:\bin\newlisp\newlisp.exe -e '(read 0 b 10000)(map char (explode b))'

Note, I don't really use windows myself, so I'm guessing about piping the output from one command (thescript) to another (the unpacking into character codes). In any case, the resulting character codes list should then begin
Code: Select all
(67 111 110 116 101 110 116 45 116 121 112 101 ...
ralph.ronnquist
 
Posts: 188
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Wed Mar 27, 2019 3:28 pm

@ralph.ronnquist
You bet that I have an Hex editor!! I don't see any garbage - which puzzles me. I've included a screen capture of the output. Now I'm wondering what Apache is choking on!! Pain in the Pa-toot!! LOL
Attachments
outputData1.png
outputData1.png (28.68 KiB) Viewed 420 times
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby Lutz » Wed Mar 27, 2019 4:55 pm

Check how NEWLISPDIR and PROGRAMFILES are defined.

See here for an explanation of these directories: http://www.newlisp.org/downloads/newlis ... ml#startup

You could use the fowlowing script to check:

(println (env "PROGRAMFILES")) ; only defined on Windows
(println (env "NEWLISPDIR"))
(exit)
Lutz
 
Posts: 5269
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Wed Mar 27, 2019 5:28 pm

Both are set. I've installed newlisp in c:\bin\newlisp to make sure that I did not have future issues with spaces in directory names like "Program Files" etc. So NEWLISPDIR points to c:\bin\newlisp.

PROGRAMFILES is set to "Program Files". I changed it to c:\bin\newlisp.

The change did not fix my problem. BTW what did you do in 2015 to fix this very same issue? You modified something in newlisp.exe and asked me to try your modifications. One of them worked!!! :-)
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby rickyboy » Wed Mar 27, 2019 5:34 pm

duke,

Hex dump seems ok to me. I'm assuming you did something like:

Code: Select all
C:\> newlisp-cgi-script > newlisp-cgi.out

Then, looked at newlisp-cgi.out in the hex editor. Maybe try also something like:

Code: Select all
C:\> perl-cgi-script > perl-cgi.out

and maybe the same with the Ruby, CLISP scripts, etc. (i.e., all the CGI scripts that Apache doesn't choke on), and see if you can spot any difference between each output from those and newlisp-cgi.out.
(λx. x x) (λx. x x)
rickyboy
 
Posts: 568
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Postby Lutz » Fri Mar 29, 2019 2:47 pm

Try the following:

Create a subdirectory "newlisp" in "Program Files" and put the newlisp.exe executable in there. If there are any modules you are going to use, create a "modules" directory inside the "newlisp" directory.

Code: Select all
(env "PROGRAMFILES") => C:\Program Files (x86)
(env "NEWLISPDIR") => C:\Program Files (x86)\newlisp

Path of newlisp.exe is now: C:\Program Files (x86)\newlisp\newlisp.exe

So both environment variables point to directories and the path in NEWLISPDIR must be in the executable path.

May be Windows finds the newlisp executable when using the command line, but not when called from Apache.

The original fix for Apache in 2015 was putting defaults (the above) for both environment variables.
Lutz
 
Posts: 5269
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Fri Mar 29, 2019 3:44 pm

@Lutz Thanks for all the trouble you went through!! Unfortunately, after doing all your recommendations, nothing changed! No Cigars!! I still get the same response from Apache:

[Fri Mar 29 09:35:53.040187 2019] [http:error] [pid 10048:tid 1076] [client 127.0.0.1:51447] AH02429: Response header name 'Environment variable TMP not set, assuming /tmp .Content-Type' contains invalid characters, aborting request


I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby rickyboy » Fri Mar 29, 2019 7:21 pm

dukester wrote:I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?

That’s not what your hex dump indicates.

It may be due to a configuration/interface issue — I believe that’s the trail Lutz was going down and a good one to pursue.

It would be nice to see what text stream Apache is seeing. It may be something very unlike what we are imagining in our heads. Any way to do that? IOW, does Apache have a better way to diagnose these problems (like a debug level) because that exception message that you get tells us hardly anything at all and it may even be “lying” to us.

Also, if you don’t know already, you may want to research into how Apache is calling out to / running your script. There may be some devil in the details there. If they didn’t doc it well, you might have to go code spelunking. Ugh.
(λx. x x) (λx. x x)
rickyboy
 
Posts: 568
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Fri Mar 29, 2019 10:06 pm

@rickyboy You know - I'm thinking that Lutz has already wasted enough time looking into this issue. If there were hundreds of folks using newlisp on Apache, it might then be a worthwhile investment of time. I'm sure that the folks at Apache would pretty much feel the same way! How many people are they aware of that use Apache to run newlisp scripts. I'm sure they couldn't be bothered.

How's the song go? "You got to know when to hold 'em, know when to fold 'em, Know when to walk away and know when to run. " Maybe that time is come. I'll pick at it now and then, but let's not waste anymore time. I thought the solution might be simple, and already solved. But there I went thinking again! LOL Thanks anyway! Everybody ....
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Re: Newlisp-Apache-CGI Issues Again

Postby rickyboy » Fri Mar 29, 2019 11:09 pm

duke, I wasn't saying for Lutz to look at it, but for *you* to look at it. lol!

No worries, I get it. I've been there many times. We do indeed have to make that determination: whether to spelunk or not. :) Sometimes, it's not worth it, especially when we have time constraints. I hope you get some time here and there to mess around with it ... and succeed!

All the best! rickyboy
(λx. x x) (λx. x x)
rickyboy
 
Posts: 568
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: Newlisp-Apache-CGI Issues Again

Postby dukester » Fri Mar 29, 2019 11:27 pm

Yeah! I realized that you meant me! I was just trying to be civil and say "No $%%^^* way!!" LOL And Lutz does have better things to do! Thanks - I will fiddle with it as and when the mood hits me. Good luck yourself!
duke
dukester
 
Posts: 108
Joined: Tue May 08, 2007 1:06 pm
Location: Alberta, Canada

Next

Return to newLISP and the O.S.

Who is online

Users browsing this forum: No registered users and 0 guests

cron