I/O multiplexing on pipes

Q&A's, tips, howto's
Post Reply
jef
Posts: 11
Joined: Sun May 18, 2014 8:42 am

I/O multiplexing on pipes

Post by jef »

Had to perform some I/O multiplexing on pipes (read multiple pipes in parallel
until they're closed), and realized that newlisp does not provide a function like
"select" in C to monitor file descriptors.

I've looked in the documentation but didn't find any solution. I've ended doing a C
implementation to get the desired result, something like that (select_readable is a
wrapper calling my c implementation, taking a list of file descriptors and returning
the first one that is ready to read):

Code: Select all

(import "./mylib.so" "c_select_readable")
...
; fd1 and fd2, two filedescriptors I'd like to read
(setq fds (list fd1 fd2))
(while (and fds (setq fd (select_readable fds)))
  (if (nil? (read-line fd))
    (replace fd fds)
    (println current-line)))
If you have any 100% newlisp solution, I would be interested. The peek function
might have been a candidate, but is not enough, as I need to know in a non
blocking way when a pipe has been closed.

PS: as a side note, while coding my C extension, I realized I wasn't able to get a signed
integer values as a return value, even if I mention "int" as the str-return-type during import:

Code: Select all

$ cat returnneg.c
int returnneg()
{
        return -1;
}

$ gcc -Wall -fPIC returnneg.c -shared -o returnneg.so

> (import "./returnneg.so" "returnneg" "int")
returnneg@7F1E4B1D2645
> (returnneg)
4294967295
Any idea of what I'm doing wrong? Thanks in advance.

Lutz
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: I/O multiplexing on pipes

Post by Lutz »

What OS and newLISP are you using? Is your newLISP 32bit or 64bit and does it say "ffilib" in the banner? If it does not say "ffilib" in the banner, it is only compiled for the simple FFI which ignores type specs in the import statement. Only with "ffilib" in the banner it is compiled for the extended FFI described in the reference manual.

There are testcases for int sign extension in newlisp-x.x.x/util/ffitest.c and newlisp-x.x.x/qa-specific-tests/qa-libffi which do work on 32bit and 64bit Unix and 32bit Windows.

For non-blocking pipes try to create named pipes using open with the "non-block" option (only on Unix), then use peek, which will return nil on closed pipe handles.

jef
Posts: 11
Joined: Sun May 18, 2014 8:42 am

Re: I/O multiplexing on pipes

Post by jef »

Thanks for the advice about named pipes and peek.

Code has been tested on Debian/Squeeze and Debian/Jessie, on amd64.
Newlisp is compiled with ffilib.

Code: Select all

$ gcc -m64 -fPIC ffitest.c -shared -o ffitest.dylib
$ newlisp
newLISP v.10.6.0 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (sys-info)
(436 576460752303423488 408 1 0 2048 0 4355 10600 1409)
> (import "./ffitest.dylib" "ret_sint32" "int")
ret_sint32@7F3325FA3C1B
> (ret_sint32 -1)
4294967295

Lutz
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: I/O multiplexing on pipes

Post by Lutz »

Please try the following:

Code: Select all

(import "./ffitest.dylib" "ret_sint32" "int" "int")

(ret_sint32 -1)  

(ret_sint32 4294967295)
Note, that the import statement has two times "int", the first for the return type of the function and the second "int" for the argument to the function. Both calls to ret_sint32 should yield -1.

jef
Posts: 11
Joined: Sun May 18, 2014 8:42 am

Re: I/O multiplexing on pipes

Post by jef »

Yes your code works for the ret_sint32 function, which takes an int as an argument and return an int as a result. My problems comes when I want to call a function taking no argument and returning a signed int:

Code: Select all

$ cat returnneg.c
int returnneg()
{
        return -1;
}

$ gcc -Wall -fPIC returnneg.c -shared -o returnneg.so

$ newlisp
newLISP v.10.6.0 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (import "./returnneg.so" "returnneg" "int")
returnneg@7F1E4B1D2645
> (returnneg)
4294967295
After having a look at newlisp source code in nl-import.c, I realize that
libffi is not used if str-param-type is empty (nl-import.c at lines
148-149 for Unix platforms). So, I have to cheat a bit and make my call
this way to get the expected result:

Code: Select all

> (import "./returnneg.so" "returnneg" "int" "void")
returnneg@7F1E4B1D2645
> (returnneg)
-1

Astrobe
Posts: 39
Joined: Mon Jan 11, 2010 9:41 pm

Re: I/O multiplexing on pipes

Post by Astrobe »

int returnneg(void) is a valid prototype.It's actually "more correct" than int returnneg(): the former tells that the function takes no arguments, while the latter can take any number of arguments.

source: http://www.drdobbs.com/c-and-c-case-stu ... /184401562

ralph.ronnquist
Posts: 216
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: I/O multiplexing on pipes

Post by ralph.ronnquist »

How about net-select?
http://www.newlisp.org/downloads/newlis ... net-select
It's documented to deal with sockets, but at least on linux this means file descriptors in general (implemented at nl-sock.c#p_netSelect).

Code: Select all

(net-select (list fd1 fd2) "read" -1)
This doesn't beat the pleasure of attaching C code of your own, but it does seem to offer the functionality.

jef
Posts: 11
Joined: Sun May 18, 2014 8:42 am

Re: I/O multiplexing on pipes

Post by jef »

ralph.ronnquist wrote:How about net-select?.
Yes Ralph, you're totally right, it's exactly what I was looking for.
Works perfectly.

Post Reply