cgi; stream output from exe to stdout

Q&A's, tips, howto's

cgi; stream output from exe to stdout

Postby hds1 » Sat Mar 29, 2014 8:10 am

Hello all,

in a cgi script i'am trying to pipe the output from an exec prog via a stream to the browser. i.e. the output from "mapserver" et al. ; could be MByte large pictures.
In Perl i can use print and backquotes like:
<snip>
$ENV{"QUERY_STRING"}=blah blah
print `/path/to/mapserv`; # -> output goes to the browser
<snap>

I cannot wrap my mind around the exec statement in newlisp. According to the docs i could use:
(exec "/path/to/mapserv" query) opening a str-input.
But IMHO that store's the content in "query" which i would like to avoid.

How can i redirect the stream or "query" directly to "stdout" ?

Regards
hds1
hds1
 
Posts: 27
Joined: Thu Mar 20, 2014 5:02 pm

Re: cgi; stream output from exe to stdout

Postby Lutz » Sat Mar 29, 2014 5:57 pm

Code: Select all
#!/usr/bin/env newlisp

(print  "Content-Type: text/html\r\n\r\n")

(println "<pre>")
(! "/bin/ls -ltr downloads/" )
(println "</pre>")

(dolist (line (exec "/bin/cat downloads/SHA1.txt"))
        (println line "<br/>")
)

(exit)

using exec you can filter or modify the output.

try it here: http://www.newlisp.org/example.cgi
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby Lutz » Sat Mar 29, 2014 11:04 pm

http://www.newlisp.org is currently - UTC 23:02, 4:02pm PDT - down and the service provider will restore from an older backup. This backup will probably not contain the newlisp.org/example.cgi from the last post. I will put it back as soon as I have a chance.

PS: site is up again at 8:08pm PDT and example works.
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Sun Mar 30, 2014 4:25 pm

Lutz, how does that work if it is a process that runs forever, perhaps "tail -f /var/www/log/mywebsite.com" ?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: cgi; stream output from exe to stdout

Postby hds1 » Sun Mar 30, 2014 6:09 pm

@Lutz,

thanks for the reply, but still the output is saved first in avariable and then send out to whom ever it shall go.

Can't it be done without interime saving ?

Consider Pictures, video streams or such. You don't want to save them, you want to pass the stream "through".

Thanks
hds1
hds1
 
Posts: 27
Joined: Thu Mar 20, 2014 5:02 pm

Re: cgi; stream output from exe to stdout

Postby Lutz » Sun Mar 30, 2014 6:35 pm

see here: http://www.newlisp.org/downloads/CodePa ... html#toc-2

sub chapter Scripts as pipes
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Sun Mar 30, 2014 6:40 pm

Lutz wrote:see here: http://www.newlisp.org/downloads/CodePa ... html#toc-2

sub chapter Scripts as pipes


Ah, right, thank you Lutz.

To the original commenter: the answer is to use the (pipe) and (process) functions instead of (exec). The link Lutz posted above has a good example of the design pattern for doing what you want to do.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: cgi; stream output from exe to stdout

Postby Lutz » Sun Mar 30, 2014 6:59 pm

and this link may also be relevant:

http://www.newlisp.org/downloads/CodePa ... tml#toc-15
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Mon Mar 31, 2014 5:14 pm

I thought I'd help out by making a "popen" and "pclose" function, just like in Richard Stevens "Advanced Unix Programming". But my example code doesn't work. I'm puzzled. It should work. The popen function returns exactly the data I expect it to; the pid and a pair of file descriptors for reading and writing. Lutz, what am I doing wrong?

Code: Select all
#!/usr/bin/newlisp

(define (popen a)
  (let (inp (pipe) outp (pipe))
    (list
      (process a (first outp) (last inp))
      (first inp)
      (last outp)
    )
  )
)

(define (pclose p)
  (close (p 1))
  (close (p 2))
  (wait-pid (p 0))
)

(letn (proc (popen "echo hello")
       ppid (proc 0)
       pin  (proc 1)
       pout (proc 2))
  (while (read-line pin)
    (write-line)
  )
  (pclose proc)
)

(exit)


Using the popen and pclose functions makes doing this pipe/filter/exec thing really simple.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: cgi; stream output from exe to stdout

Postby Lutz » Mon Mar 31, 2014 7:02 pm

here is another working example using bc:

Code: Select all
#!/usr/bin/newlisp

(map set '(myin bcout) (pipe))
(map set '(bcin myout) (pipe))

(process "/usr/bin/bc" bcin bcout)

(until (= "quit" (read-line))
    (write-line myout)
    (println "=> " (read-line myin))
)

(exit)

the program is called bc-pipe
Code: Select all
~> ./bc-pipe
6 * 7
=> 42
quit
~>


you also can use (peek myin) to check for the number of chars available on the input pipe.

ps: bc is a Unix calculator utility
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Mon Mar 31, 2014 8:09 pm

Yes, I based on my code on your bc example.

After trying everything, I finally put an absolute pathname to the binary. I didn't realize that (process) doesn't search the $PATH variable for the binary to execute. Once I fixed that, it worked as expected.

So, now that it works, I've cleaned up the popen and pclose functions, fixed a file descriptor leak in popen. I think these functions are useful enough I wish they could be part of the default base system somehow.

Here is the updated code, I hope the person who started this thread finds it useful, I know I will.

Code: Select all
#!/usr/bin/newlisp

(define (popen a)
  (letn (inp (pipe) outp (pipe) pidp (process a (first outp) (last inp)))
    (close (first outp))
    (close (last inp))
    (list pidp (first inp) (last outp))
  )
)

(define (pclose p)
  (close (p 1))
  (close (p 2))
  (wait-pid (p 0))
)

;; example of how to use the popen and pclose functions
(setq proc (popen "/bin/ls -l"))
(while (read-line (proc 1))
  (write-line)
)
(pclose proc)
(exit)
;; end of example


In fact, with the popen/pclose functions, it should be pretty easy to implement a Unix command shell with file io redirection, etc.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: cgi; stream output from exe to stdout

Postby Lutz » Mon Mar 31, 2014 9:06 pm

there is a pattern of real-path to find the executable path:

Code: Select all
> (real-path "echo" true)
"/bin/echo"
> (real-path "bc" true)
"/usr/bin/bc"
>
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Mon Mar 31, 2014 9:24 pm

Lutz wrote:there is a pattern of real-path to find the executable path:

Code: Select all
> (real-path "echo" true)
"/bin/echo"
> (real-path "bc" true)
"/usr/bin/bc"
>


Very nice.

Would it be possible to update the (device) function? Implementing popen and pclose, I notice that (device) assumes that the same file handle will be used for read and write. This makes it not very helpful for connecting up to a (process).

Could we update the syntax so that (device n1) works the same as now, but (device n1 n2) sets n1 as the stdin, and n2 sets the stdout? I don't know of anything in newlisp that writes to stderr, or I'd suggest (device n1 n2 n3) as well, with n3 becoming the new stderr. And also the syntax of "nil" if you only want to clobber stdout or stderr and leave the others alone. Ie, (device n1 nil n3) would reset stdin and stderr but leave stdout alone. And (device nil) would reset everything to normal 0 stdin 1 stdout 2 stderr, as per the POSIX spec. I think even Microsoft and Apple respect this?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: cgi; stream output from exe to stdout

Postby Lutz » Wed Apr 02, 2014 1:22 pm

Looks like a good idea, but wouldn't be for 10.6.0, also with a different name. The return value would be different - a list - and I don't want to break compatibility with older code. device is one of the oldest functions in newLISP.
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Wed Apr 02, 2014 4:13 pm

Lutz wrote:Looks like a good idea, but wouldn't be for 10.6.0, also with a different name. The return value would be different - a list - and I don't want to break compatibility with older code. device is one of the oldest functions in newLISP.


For the current syntax (device n), the return value could stay the same as it is currently, the (list) return val would only be for the extended syntax, (device n1 n2 [n3])

Since you are doing a major point release, isn't this the best time to put a new function in? Or would you rather put new functionality in during the minor point releases?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: cgi; stream output from exe to stdout

Postby Lutz » Wed Apr 02, 2014 4:30 pm

What I meant was the form (device) to inquire the current channel, which returns one integer. I would rather introduce it in development releases, as the the whole std I/O area touches many other areas, e.g. web and cgi.
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: cgi; stream output from exe to stdout

Postby TedWalther » Wed Apr 02, 2014 6:42 pm

Ok.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 605
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC


Return to newLISP in the real world

Who is online

Users browsing this forum: No registered users and 1 guest