cgi; stream output from exe to stdout

[hds1]

Hello all,

in a cgi script i'am trying to pipe the output from an exec prog via a stream to the browser. i.e. the output from "mapserver" et al. ; could be MByte large pictures.
In Perl i can use print and backquotes like:
<snip>
$ENV{"QUERY_STRING"}=blah blah
print `/path/to/mapserv`; # -> output goes to the browser
<snap>

I cannot wrap my mind around the exec statement in newlisp. According to the docs i could use:
(exec "/path/to/mapserv" query) opening a str-input.
But IMHO that store's the content in "query" which i would like to avoid.

How can i redirect the stream or "query" directly to "stdout" ?

Regards
hds1

[Lutz]

Code: Select all

#!/usr/bin/env newlisp

(print  "Content-Type: text/html\r\n\r\n")

(println "<pre>")
(! "/bin/ls -ltr downloads/" )
(println "</pre>")

(dolist (line (exec "/bin/cat downloads/SHA1.txt"))
        (println line "<br/>")
)

(exit)

using exec you can filter or modify the output.

try it here: http://www.newlisp.org/example.cgi

[Lutz]

http://www.newlisp.org is currently - UTC 23:02, 4:02pm PDT - down and the service provider will restore from an older backup. This backup will probably not contain the newlisp.org/example.cgi from the last post. I will put it back as soon as I have a chance.

PS: site is up again at 8:08pm PDT and example works.

[TedWalther]

Lutz, how does that work if it is a process that runs forever, perhaps "tail -f /var/www/log/mywebsite.com" ?

[hds1]

@Lutz,

thanks for the reply, but still the output is saved first in avariable and then send out to whom ever it shall go.

Can't it be done without interime saving ?

Consider Pictures, video streams or such. You don't want to save them, you want to pass the stream "through".

Thanks
hds1

[Lutz]

see here: http://www.newlisp.org/downloads/CodePa ... html#toc-2

sub chapter Scripts as pipes

[TedWalther]

see here: http://www.newlisp.org/downloads/CodePa ... html#toc-2

sub chapter Scripts as pipes

Ah, right, thank you Lutz.

To the original commenter: the answer is to use the (pipe) and (process) functions instead of (exec). The link Lutz posted above has a good example of the design pattern for doing what you want to do.

[Lutz]

and this link may also be relevant:

http://www.newlisp.org/downloads/CodePa ... tml#toc-15

[TedWalther]

I thought I'd help out by making a "popen" and "pclose" function, just like in Richard Stevens "Advanced Unix Programming". But my example code doesn't work. I'm puzzled. It should work. The popen function returns exactly the data I expect it to; the pid and a pair of file descriptors for reading and writing. Lutz, what am I doing wrong?

Code: Select all

#!/usr/bin/newlisp

(define (popen a)
  (let (inp (pipe) outp (pipe))
    (list
      (process a (first outp) (last inp))
      (first inp)
      (last outp)
    )
  )
)

(define (pclose p)
  (close (p 1))
  (close (p 2))
  (wait-pid (p 0))
)

(letn (proc (popen "echo hello")
       ppid (proc 0)
       pin  (proc 1)
       pout (proc 2))
  (while (read-line pin)
    (write-line)
  )
  (pclose proc)
)

(exit)

Using the popen and pclose functions makes doing this pipe/filter/exec thing really simple.

[Lutz]

here is another working example using bc:

Code: Select all

#!/usr/bin/newlisp

(map set '(myin bcout) (pipe))
(map set '(bcin myout) (pipe))

(process "/usr/bin/bc" bcin bcout)

(until (= "quit" (read-line))
    (write-line myout)
    (println "=> " (read-line myin))
)

(exit)

the program is called bc-pipe

Code: Select all

~> ./bc-pipe 
6 * 7
=> 42
quit
~> 

you also can use (peek myin) to check for the number of chars available on the input pipe.

ps: bc is a Unix calculator utility

[TedWalther]

Yes, I based on my code on your bc example.

After trying everything, I finally put an absolute pathname to the binary. I didn't realize that (process) doesn't search the $PATH variable for the binary to execute. Once I fixed that, it worked as expected.

So, now that it works, I've cleaned up the popen and pclose functions, fixed a file descriptor leak in popen. I think these functions are useful enough I wish they could be part of the default base system somehow.

Here is the updated code, I hope the person who started this thread finds it useful, I know I will.

Code: Select all

#!/usr/bin/newlisp

(define (popen a)
  (letn (inp (pipe) outp (pipe) pidp (process a (first outp) (last inp)))
    (close (first outp))
    (close (last inp))
    (list pidp (first inp) (last outp))
  )
)

(define (pclose p)
  (close (p 1))
  (close (p 2))
  (wait-pid (p 0))
)

;; example of how to use the popen and pclose functions
(setq proc (popen "/bin/ls -l"))
(while (read-line (proc 1))
  (write-line)
)
(pclose proc)
(exit)
;; end of example

In fact, with the popen/pclose functions, it should be pretty easy to implement a Unix command shell with file io redirection, etc.

[Lutz]

there is a pattern of real-path to find the executable path:

Code: Select all

> (real-path "echo" true)
"/bin/echo"
> (real-path "bc" true)
"/usr/bin/bc"
> 

[TedWalther]

there is a pattern of real-path to find the executable path:

Code: Select all

> (real-path "echo" true)
"/bin/echo"
> (real-path "bc" true)
"/usr/bin/bc"
> 

Very nice.

Would it be possible to update the (device) function? Implementing popen and pclose, I notice that (device) assumes that the same file handle will be used for read and write. This makes it not very helpful for connecting up to a (process).

Could we update the syntax so that (device n1) works the same as now, but (device n1 n2) sets n1 as the stdin, and n2 sets the stdout? I don't know of anything in newlisp that writes to stderr, or I'd suggest (device n1 n2 n3) as well, with n3 becoming the new stderr. And also the syntax of "nil" if you only want to clobber stdout or stderr and leave the others alone. Ie, (device n1 nil n3) would reset stdin and stderr but leave stdout alone. And (device nil) would reset everything to normal 0 stdin 1 stdout 2 stderr, as per the POSIX spec. I think even Microsoft and Apple respect this?

[Lutz]

Looks like a good idea, but wouldn't be for 10.6.0, also with a different name. The return value would be different - a list - and I don't want to break compatibility with older code. device is one of the oldest functions in newLISP.

[TedWalther]

Looks like a good idea, but wouldn't be for 10.6.0, also with a different name. The return value would be different - a list - and I don't want to break compatibility with older code. device is one of the oldest functions in newLISP.

For the current syntax (device n), the return value could stay the same as it is currently, the (list) return val would only be for the extended syntax, (device n1 n2 [n3])

Since you are doing a major point release, isn't this the best time to put a new function in? Or would you rather put new functionality in during the minor point releases?

[Lutz]

What I meant was the form (device) to inquire the current channel, which returns one integer. I would rather introduce it in development releases, as the the whole std I/O area touches many other areas, e.g. web and cgi.

[TedWalther]

Ok.