passing pipe to process?

[nigelbrown]

In the ref man it says:
syntax: (pipe)
pipe creates an interprocess communications pipe and returns the read and write handles
to it in a list. This function is only available on Linux/BSD compiles and not working on the
Win32 or CYGWIN versions.
(pipe) => (3 4) ; 3 for read, 4 for writing
The pipe handles can be passed on to a child process launched via process for inter process
communications.

QUESTION: How are the pipe handles passed to process (on linux of course)?

The man gives for process:
process
syntax: (process str−command)
process works similar to ! but in a non−blocking fashion, launching a child−process
specified in str−command and then returning immeadeately with true.
example:
(process "notepad") => true
See also pipe for interprocess communications.

[Lutz]

you would pass the pipe and other parameters to the process called on the command line:

(pipe) => (3 4)

(process "newlisp dothis.lsp 3 4")

This would invoke a second newLISP process and pass the 3 parameters: a file name to execute and the pipe parameters

'dothis.lsp' would read the parameters using the function (main-args):

(main-args) => ("newlisp" "dothis.lsp" "3" "4") ;; executed in dolisp

dolisp then extracts the pipe handles and could read/write something to the pipe, then the calling process could read it back etc. You read/write the pipe handles just like files.

Lutz

[nigelbrown]

Thanks Lutz.

Trying it works fine
but if my process has exited and the original process does another read-line from that
pipe it sits waiting (?forever) - is there a way of testing the pipe for data without
going into a blocking wait?

Regards
Nigel

[nigelbrown]

In answer to my own question it looks like
(read-buffer will do it?
Nigel

[nigelbrown]

No, it seems to wait too:
[nigel@p1300 newlisp_7303]$ newlisp
newLISP v7.3.3 Copyright (c) 2003 Lutz Mueller. All rights reserved.

> (setq pipes (pipe))
(3 4)
> (process "./newlisp dothis.lsp 3 4")
true
> (setq replied (read-line 3))
"hello from process"
> (setq rep2 (read-buffer 3 'buffer 10))

I'm using Mandrake 9.1

Any way of testing pipe to see if input is there?
I want to set a number of processes on potentially long tasks and check for replies.
Hopefully once I get my Quantian (http://dirk.eddelbuettel.com/quantian.html) OpenMosix cluster (home hobby cluster) running
then the processes will distribute across the cluster when running.
If I can't manage them with straight reads then I'll have to think of fancier coordination.

Nigel

[nigelbrown]

(Apologies if this is sounding more like a blog than a thread)

I will try properly/gracefully closing the pipe from the dothis.lsp end - I was
thinking that program termination would do that but perhaps not.
I'll report back once I've had access to a linux system.

Nigel

[Lutz]

I guess there would be some kind of 'ioctl' call to check for the readiness of the pipe. I wonder if the newLISP function 'net-select' would work, which does the same for network sockets.

If your application can work on Linux/UNIX than I guess that using sockets in the first place, will work for you as well? and it would be perhaps more portable?

Just coming back from a family Thanksgiving holiday in Key-West and not with much time until the weekend is over ...

Lutz

[nigelbrown]

Hi,
I found the reason that the read was blocking was that although the process had closed 4 the original newlisp still had it open! Closing 4 in the program that is reading from 3 fixed the blocking viz:
> (pipe)
(3 4)
> (process "./newlisp dothis.lsp 3 4")
true
> (close 4)
true
> (read-buffer 3 'buf 10)
about to read
<read bytes: 10 >
10
> buf
"hello from"
> (read-buffer 3 'buf 10)
about to read
<read bytes: 8 >
8
> buf
" process"
> (read-buffer 3 'buf 10)
about to read
<read bytes: 0 >
0
>

The "about to read" and "read bytes:" are from some debug printf's I used to see what was happening.
What put me on to the problem was this explaination from http://www.opengroup.org/onlinepubs/007 ... /read.html::
"When attempting to read from an empty pipe or FIFO:

If no process has the pipe open for writing, read() will return 0 to indicate end-of-file.

If some process has the pipe open for writing and O_NONBLOCK is set, read() will return -1 and set errno to [EAGAIN].

If some process has the pipe open for writing and O_NONBLOCK is clear, read() will block the calling thread until some data is written or the pipe is closed by all processes that had the pipe open for writing. "
<note last paragraph>
It also lines up with advice from the book "Linux programming by example" by Kurt Wall that points out:
"The general rule is that the reader process closes the write end of the pipe and the writer process closes the reader end of the pipe."
by the way dothis.lsp was:
(close 3)
(write-buffer 4 "hello from process" )
(close 4)
(exit)

Hope this serves as a tip for anyone else new to trying out pipes.

Nigel

[Lutz]

Thanks for the research, Nigel. I will put some of this in the manual, so people have it easier when using pipes.


Lutz

[nigelbrown]

regarding the 7.3.17 manual comment:
While read-line will block until a newline character is received, read-buffer will block when less characters than specified are available

read-buffer will not block if the pipe has been closed with less than specified bytes
which can be seen from the fragment

> (read-buffer 3 'buf 10)
about to read
<read bytes: 8 >
8
> buf
" process"

above where the last 8 bytes from the closed pipe are available
and from

> (read-buffer 3 'buf 10)
about to read
<read bytes: 0 >
0
>

where no blocking occues but zero bytes are returned as available fron the closed pipe.

Perhaps docs should read:
read-buffer will block when less characters than specified are available from a pipe that has not had the writiing end closed by all processes.

Nigel


PS I've not looked at read-line blocking at all, particularly what happens if the pipe has been closed but without a newline.

[Lutz]

I added "hat has not had the writiing end closed by all processes" in the documentation.

Lutz