cgi; stream output from exe to stdout

Q&A's, tips, howto's
Locked
hds1
Posts: 28
Joined: Thu Mar 20, 2014 5:02 pm

cgi; stream output from exe to stdout

Post by hds1 »

Hello all,

in a cgi script i'am trying to pipe the output from an exec prog via a stream to the browser. i.e. the output from "mapserver" et al. ; could be MByte large pictures.
In Perl i can use print and backquotes like:
<snip>
$ENV{"QUERY_STRING"}=blah blah
print `/path/to/mapserv`; # -> output goes to the browser
<snap>

I cannot wrap my mind around the exec statement in newlisp. According to the docs i could use:
(exec "/path/to/mapserv" query) opening a str-input.
But IMHO that store's the content in "query" which i would like to avoid.

How can i redirect the stream or "query" directly to "stdout" ?

Regards
hds1

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

Code: Select all

#!/usr/bin/env newlisp

(print  "Content-Type: text/html\r\n\r\n")

(println "<pre>")
(! "/bin/ls -ltr downloads/" )
(println "</pre>")

(dolist (line (exec "/bin/cat downloads/SHA1.txt"))
        (println line "<br/>")
)

(exit)
using exec you can filter or modify the output.

try it here: http://www.newlisp.org/example.cgi

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

http://www.newlisp.org is currently - UTC 23:02, 4:02pm PDT - down and the service provider will restore from an older backup. This backup will probably not contain the newlisp.org/example.cgi from the last post. I will put it back as soon as I have a chance.

PS: site is up again at 8:08pm PDT and example works.

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

Lutz, how does that work if it is a process that runs forever, perhaps "tail -f /var/www/log/mywebsite.com" ?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

hds1
Posts: 28
Joined: Thu Mar 20, 2014 5:02 pm

Re: cgi; stream output from exe to stdout

Post by hds1 »

@Lutz,

thanks for the reply, but still the output is saved first in avariable and then send out to whom ever it shall go.

Can't it be done without interime saving ?

Consider Pictures, video streams or such. You don't want to save them, you want to pass the stream "through".

Thanks
hds1

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

see here: http://www.newlisp.org/downloads/CodePa ... html#toc-2

sub chapter Scripts as pipes

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

Lutz wrote:see here: http://www.newlisp.org/downloads/CodePa ... html#toc-2

sub chapter Scripts as pipes
Ah, right, thank you Lutz.

To the original commenter: the answer is to use the (pipe) and (process) functions instead of (exec). The link Lutz posted above has a good example of the design pattern for doing what you want to do.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

and this link may also be relevant:

http://www.newlisp.org/downloads/CodePa ... tml#toc-15

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

I thought I'd help out by making a "popen" and "pclose" function, just like in Richard Stevens "Advanced Unix Programming". But my example code doesn't work. I'm puzzled. It should work. The popen function returns exactly the data I expect it to; the pid and a pair of file descriptors for reading and writing. Lutz, what am I doing wrong?

Code: Select all

#!/usr/bin/newlisp

(define (popen a)
  (let (inp (pipe) outp (pipe))
    (list
      (process a (first outp) (last inp))
      (first inp)
      (last outp)
    )
  )
)

(define (pclose p)
  (close (p 1))
  (close (p 2))
  (wait-pid (p 0))
)

(letn (proc (popen "echo hello")
       ppid (proc 0)
       pin  (proc 1)
       pout (proc 2))
  (while (read-line pin)
    (write-line)
  )
  (pclose proc)
)

(exit)
Using the popen and pclose functions makes doing this pipe/filter/exec thing really simple.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

here is another working example using bc:

Code: Select all

#!/usr/bin/newlisp

(map set '(myin bcout) (pipe))
(map set '(bcin myout) (pipe))

(process "/usr/bin/bc" bcin bcout)

(until (= "quit" (read-line))
    (write-line myout)
    (println "=> " (read-line myin))
)

(exit)
the program is called bc-pipe

Code: Select all

~> ./bc-pipe 
6 * 7
=> 42
quit
~> 
you also can use (peek myin) to check for the number of chars available on the input pipe.

ps: bc is a Unix calculator utility

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

Yes, I based on my code on your bc example.

After trying everything, I finally put an absolute pathname to the binary. I didn't realize that (process) doesn't search the $PATH variable for the binary to execute. Once I fixed that, it worked as expected.

So, now that it works, I've cleaned up the popen and pclose functions, fixed a file descriptor leak in popen. I think these functions are useful enough I wish they could be part of the default base system somehow.

Here is the updated code, I hope the person who started this thread finds it useful, I know I will.

Code: Select all

#!/usr/bin/newlisp

(define (popen a)
  (letn (inp (pipe) outp (pipe) pidp (process a (first outp) (last inp)))
    (close (first outp))
    (close (last inp))
    (list pidp (first inp) (last outp))
  )
)

(define (pclose p)
  (close (p 1))
  (close (p 2))
  (wait-pid (p 0))
)

;; example of how to use the popen and pclose functions
(setq proc (popen "/bin/ls -l"))
(while (read-line (proc 1))
  (write-line)
)
(pclose proc)
(exit)
;; end of example
In fact, with the popen/pclose functions, it should be pretty easy to implement a Unix command shell with file io redirection, etc.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

there is a pattern of real-path to find the executable path:

Code: Select all

> (real-path "echo" true)
"/bin/echo"
> (real-path "bc" true)
"/usr/bin/bc"
> 

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

Lutz wrote:there is a pattern of real-path to find the executable path:

Code: Select all

> (real-path "echo" true)
"/bin/echo"
> (real-path "bc" true)
"/usr/bin/bc"
> 
Very nice.

Would it be possible to update the (device) function? Implementing popen and pclose, I notice that (device) assumes that the same file handle will be used for read and write. This makes it not very helpful for connecting up to a (process).

Could we update the syntax so that (device n1) works the same as now, but (device n1 n2) sets n1 as the stdin, and n2 sets the stdout? I don't know of anything in newlisp that writes to stderr, or I'd suggest (device n1 n2 n3) as well, with n3 becoming the new stderr. And also the syntax of "nil" if you only want to clobber stdout or stderr and leave the others alone. Ie, (device n1 nil n3) would reset stdin and stderr but leave stdout alone. And (device nil) would reset everything to normal 0 stdin 1 stdout 2 stderr, as per the POSIX spec. I think even Microsoft and Apple respect this?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

Looks like a good idea, but wouldn't be for 10.6.0, also with a different name. The return value would be different - a list - and I don't want to break compatibility with older code. device is one of the oldest functions in newLISP.

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

Lutz wrote:Looks like a good idea, but wouldn't be for 10.6.0, also with a different name. The return value would be different - a list - and I don't want to break compatibility with older code. device is one of the oldest functions in newLISP.
For the current syntax (device n), the return value could stay the same as it is currently, the (list) return val would only be for the extended syntax, (device n1 n2 [n3])

Since you are doing a major point release, isn't this the best time to put a new function in? Or would you rather put new functionality in during the minor point releases?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: cgi; stream output from exe to stdout

Post by Lutz »

What I meant was the form (device) to inquire the current channel, which returns one integer. I would rather introduce it in development releases, as the the whole std I/O area touches many other areas, e.g. web and cgi.

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: cgi; stream output from exe to stdout

Post by TedWalther »

Ok.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Locked