Page 1 of 2
Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 1:28 pm
by dukester
PS C:\bin\newlisp> newlisp -v
newLISP v.10.7.1 64-bit on Windows IPv4/6 UTF-8 libffi.
Running Apache2.4 on a Win10 box.
Apache is choking - once again - on the header sent by newlisp:
Content-type' contains invalid characters, aborting request
Lutz fixed a similar issue back in 2015, but it seems to have re-surfaced!
This is the code used then and now:
Code: Select all
#!C:\bin\newlisp\newlisp.exe
(print "Content-type: text/html\n\n")
(print "<html>")
(print "<h4>CGI by newLISP v" (sys-info -2)" on " ostype "</h4>")
(dolist (e (env)) (println (e 0) " : " (e 1) "</br>"))
(println "</html>")
(exit)
Anybody know how to fix this issue?
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 6:56 pm
by rickyboy
Looks suspiciously like the ole windoze versus unix line ending issue/difference. I’d check that before moving on to anything else.
For instance, what about trying this instead? (Sorry, “I don’t do windows.” :)
Code: Select all
(println "Content-type: text/html")
(println "")
Or something like that? Good luck!
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 8:47 pm
by dukester
rickyboy wrote:Looks suspiciously like the ole windoze versus unix line ending issue/difference. I’d check that before moving on to anything else.
Nope! It's a newlisp issue - just like back in 2015.
I tried your suggestion, even though I knew it wouldn't work. And it didn't. Lutz will have to get involved I'm sure. Thanks anyway!
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 9:40 pm
by ralph.ronnquist
You might want to try the initial line
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 9:55 pm
by dukester
ralph.ronnquist wrote:You might want to try the initial line
Don't know what you mean!
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 10:46 pm
by ralph.ronnquist
afair, a CGI response that includes headers should have a very first line being the HTTP response status code, then the header lines, then an empty line, and then the payload.
You may omit the headers part and start with the empty line, and then the CGI runner would insert a standard header component (with a default status line).
But not all CGI runners insert a status line before headers when headers are given. For example, newlisp does insert a default status line and default headers (in its -http mode CGI runner), but possibly apache2 doesn't.
Though, I do this irregularly enough that I typically have to refer to the CGI spec's to remind myself about the nitty-gritty, and apache2 is (like most) a moving target anyhow.
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 11:02 pm
by dukester
@ralph.ronnquist - I'm sure that you are absolutely correct. However, I've never had to do what you are suggesting. Most of my experience has been on Apache using from CL to Perl to crank out CGI stuff. Newlisp is the only interpreter that has ever given me fits with these type of issues - given that I didn't trash that very important first line of code.
Lutx fixed this issue once. He may not want to deal with it again. I've got the very same script running just fine using CLISP2.49. I don't use -http unless I want to serve an Intranet.
thx for getting back to me.
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 11:10 pm
by rickyboy
What ralph said. Apache 2.4 got "strict" with the header validations to mitigate vulnerabilities when used as an http proxy, e.g. this one in 2016 (interesting read too):
https://httpd.apache.org/security/vulne ... -2016-8743
Notice the switch where you can turn that off (although not recommended):
http://httpd.apache.org/docs/2.4/mod/co ... coloptions
Could that have been the reason that that code worked recently and now it doesn't? (Switch got changed?)
I think keeping that setting to the default strict mode and going down the path that ralph suggested for your troubleshooting sounds good to me -- at least in the meantime before you hear anything from Lutz. Again, good luck.
(If I had the same stack, I could try to recreate the problem; however. along with not being a Windows user, I'm also not an Apache user. :)
Re: Newlisp-Apache-CGI Issues Again
Posted: Tue Mar 26, 2019 11:41 pm
by dukester
@rickyboy All good stuff! But if Apache was being a bad-ass, it would be doing so with my Perl, Python and Ruby scripts! Even old CLISP scripts run well. Get where I'm going with this? ;-)
Thanks for the heads-up regarding Apache security stuff!
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 2:42 am
by xytroxon
I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.
Change the line:
To:
-- xytroxon
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 3:04 am
by dukester
[quote="xytroxon"]I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.
Change the line:
To:
Thx! I'll give that a shot!!
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 3:12 am
by dukester
Nope! No cigar! LOL
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 3:21 am
by Lutz
Try:
Code: Select all
(print "Content-Type: text/html\r\n\r\n")
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 10:35 am
by dukester
@lutz Thx, but that does not work either. Sorry!
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 12:42 pm
by ralph.ronnquist
Have you verified the byte sequence from the script, eg by running it manually and pipe its output into a file, which you the inspect with a hex code viewer? Or even perhaps like the following:
Code: Select all
% thescript | C:\bin\newlisp\newlisp.exe -e '(read 0 b 10000)(map char (explode b))'
Note, I don't really use windows myself, so I'm guessing about piping the output from one command (thescript) to another (the unpacking into character codes). In any case, the resulting character codes list should then begin
Code: Select all
(67 111 110 116 101 110 116 45 116 121 112 101 ...
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 3:28 pm
by dukester
@ralph.ronnquist
You bet that I have an Hex editor!! I don't see any garbage - which puzzles me. I've included a screen capture of the output. Now I'm wondering what Apache is choking on!! Pain in the Pa-toot!! LOL
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 4:55 pm
by Lutz
Check how NEWLISPDIR and PROGRAMFILES are defined.
See here for an explanation of these directories:
http://www.newlisp.org/downloads/newlis ... ml#startup
You could use the fowlowing script to check:
(println (env "PROGRAMFILES")) ; only defined on Windows
(println (env "NEWLISPDIR"))
(exit)
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 5:28 pm
by dukester
Both are set. I've installed newlisp in c:\bin\newlisp to make sure that I did not have future issues with spaces in directory names like "Program Files" etc. So NEWLISPDIR points to c:\bin\newlisp.
PROGRAMFILES is set to "Program Files". I changed it to c:\bin\newlisp.
The change did not fix my problem. BTW what did you do in 2015 to fix this very same issue? You modified something in newlisp.exe and asked me to try your modifications. One of them worked!!! :-)
Re: Newlisp-Apache-CGI Issues Again
Posted: Wed Mar 27, 2019 5:34 pm
by rickyboy
duke,
Hex dump seems ok to me. I'm assuming you did something like:
Code: Select all
C:\> newlisp-cgi-script > newlisp-cgi.out
Then, looked at newlisp-cgi.out in the hex editor. Maybe try also something like:
Code: Select all
C:\> perl-cgi-script > perl-cgi.out
and maybe the same with the Ruby, CLISP scripts, etc. (i.e., all the CGI scripts that Apache doesn't choke on), and see if you can spot any difference between each output from those and newlisp-cgi.out.
Re: Newlisp-Apache-CGI Issues Again
Posted: Fri Mar 29, 2019 2:47 pm
by Lutz
Try the following:
Create a subdirectory "newlisp" in "Program Files" and put the newlisp.exe executable in there. If there are any modules you are going to use, create a "modules" directory inside the "newlisp" directory.
Code: Select all
(env "PROGRAMFILES") => C:\Program Files (x86)
(env "NEWLISPDIR") => C:\Program Files (x86)\newlisp
Path of newlisp.exe is now: C:\Program Files (x86)\newlisp\newlisp.exe
So both environment variables point to directories and the path in NEWLISPDIR must be in the executable path.
May be Windows finds the newlisp executable when using the command line, but not when called from Apache.
The original fix for Apache in 2015 was putting defaults (the above) for both environment variables.
Re: Newlisp-Apache-CGI Issues Again
Posted: Fri Mar 29, 2019 3:44 pm
by dukester
@Lutz Thanks for all the trouble you went through!! Unfortunately, after doing
all your recommendations, nothing changed! No Cigars!! I still get the same response from Apache:
[Fri Mar 29 09:35:53.040187 2019] [http:error] [pid 10048:tid 1076] [client 127.0.0.1:51447] AH02429: Response header name 'Environment variable TMP not set, assuming /tmp .Content-Type' contains invalid characters, aborting request
I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?
Re: Newlisp-Apache-CGI Issues Again
Posted: Fri Mar 29, 2019 7:21 pm
by rickyboy
dukester wrote:I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?
That’s not what your hex dump indicates.
It may be due to a configuration/interface issue — I believe that’s the trail Lutz was going down and a good one to pursue.
It would be nice to see what text stream Apache is seeing. It may be something very unlike what we are imagining in our heads. Any way to do that? IOW, does Apache have a better way to diagnose these problems (like a debug level) because that exception message that you get tells us hardly anything at all and it may even be “lying” to us.
Also, if you don’t know already, you may want to research into how Apache is calling out to / running your script. There may be some devil in the details there. If they didn’t doc it well, you might have to go code spelunking. Ugh.
Re: Newlisp-Apache-CGI Issues Again
Posted: Fri Mar 29, 2019 10:06 pm
by dukester
@rickyboy You know - I'm thinking that Lutz has already wasted enough time looking into this issue. If there were hundreds of folks using newlisp on Apache, it might then be a worthwhile investment of time. I'm sure that the folks at Apache would pretty much feel the same way! How many people are they aware of that use Apache to run newlisp scripts. I'm sure they couldn't be bothered.
How's the song go? "You got to know when to hold 'em, know when to fold 'em, Know when to walk away and know when to run. " Maybe that time is come. I'll pick at it now and then, but let's not waste anymore time. I thought the solution might be simple, and already solved. But there I went thinking again! LOL Thanks anyway! Everybody ....
Re: Newlisp-Apache-CGI Issues Again
Posted: Fri Mar 29, 2019 11:09 pm
by rickyboy
duke, I wasn't saying for Lutz to look at it, but for *you* to look at it. lol!
No worries, I get it. I've been there many times. We do indeed have to make that determination: whether to spelunk or not. :) Sometimes, it's not worth it, especially when we have time constraints. I hope you get some time here and there to mess around with it ... and succeed!
All the best! rickyboy
Re: Newlisp-Apache-CGI Issues Again
Posted: Fri Mar 29, 2019 11:27 pm
by dukester
Yeah! I realized that you meant me! I was just trying to be civil and say "No $%%^^* way!!" LOL And Lutz does have better things to do! Thanks - I will fiddle with it as and when the mood hits me. Good luck yourself!