I/O multiplexing on pipes

Q&A's, tips, howto's

I/O multiplexing on pipes

Postby jef » Sat May 24, 2014 3:09 pm

Had to perform some I/O multiplexing on pipes (read multiple pipes in parallel
until they're closed), and realized that newlisp does not provide a function like
"select" in C to monitor file descriptors.

I've looked in the documentation but didn't find any solution. I've ended doing a C
implementation to get the desired result, something like that (select_readable is a
wrapper calling my c implementation, taking a list of file descriptors and returning
the first one that is ready to read):

Code: Select all
(import "./mylib.so" "c_select_readable")
...
; fd1 and fd2, two filedescriptors I'd like to read
(setq fds (list fd1 fd2))
(while (and fds (setq fd (select_readable fds)))
  (if (nil? (read-line fd))
    (replace fd fds)
    (println current-line)))


If you have any 100% newlisp solution, I would be interested. The peek function
might have been a candidate, but is not enough, as I need to know in a non
blocking way when a pipe has been closed.

PS: as a side note, while coding my C extension, I realized I wasn't able to get a signed
integer values as a return value, even if I mention "int" as the str-return-type during import:

Code: Select all
$ cat returnneg.c
int returnneg()
{
        return -1;
}

$ gcc -Wall -fPIC returnneg.c -shared -o returnneg.so

> (import "./returnneg.so" "returnneg" "int")
returnneg@7F1E4B1D2645
> (returnneg)
4294967295


Any idea of what I'm doing wrong? Thanks in advance.
jef
 
Posts: 11
Joined: Sun May 18, 2014 8:42 am

Re: I/O multiplexing on pipes

Postby Lutz » Sun May 25, 2014 2:22 am

What OS and newLISP are you using? Is your newLISP 32bit or 64bit and does it say "ffilib" in the banner? If it does not say "ffilib" in the banner, it is only compiled for the simple FFI which ignores type specs in the import statement. Only with "ffilib" in the banner it is compiled for the extended FFI described in the reference manual.

There are testcases for int sign extension in newlisp-x.x.x/util/ffitest.c and newlisp-x.x.x/qa-specific-tests/qa-libffi which do work on 32bit and 64bit Unix and 32bit Windows.

For non-blocking pipes try to create named pipes using open with the "non-block" option (only on Unix), then use peek, which will return nil on closed pipe handles.
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: I/O multiplexing on pipes

Postby jef » Sun May 25, 2014 7:00 pm

Thanks for the advice about named pipes and peek.

Code has been tested on Debian/Squeeze and Debian/Jessie, on amd64.
Newlisp is compiled with ffilib.

Code: Select all
$ gcc -m64 -fPIC ffitest.c -shared -o ffitest.dylib
$ newlisp
newLISP v.10.6.0 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (sys-info)
(436 576460752303423488 408 1 0 2048 0 4355 10600 1409)
> (import "./ffitest.dylib" "ret_sint32" "int")
ret_sint32@7F3325FA3C1B
> (ret_sint32 -1)
4294967295
jef
 
Posts: 11
Joined: Sun May 18, 2014 8:42 am

Re: I/O multiplexing on pipes

Postby Lutz » Sun May 25, 2014 7:30 pm

Please try the following:

Code: Select all
(import "./ffitest.dylib" "ret_sint32" "int" "int")

(ret_sint32 -1) 

(ret_sint32 4294967295)

Note, that the import statement has two times "int", the first for the return type of the function and the second "int" for the argument to the function. Both calls to ret_sint32 should yield -1.
Lutz
 
Posts: 5279
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: I/O multiplexing on pipes

Postby jef » Sat May 31, 2014 9:58 am

Yes your code works for the ret_sint32 function, which takes an int as an argument and return an int as a result. My problems comes when I want to call a function taking no argument and returning a signed int:

Code: Select all
$ cat returnneg.c
int returnneg()
{
        return -1;
}

$ gcc -Wall -fPIC returnneg.c -shared -o returnneg.so

$ newlisp
newLISP v.10.6.0 64-bit on Linux IPv4/6 UTF-8 libffi, options: newlisp -h

> (import "./returnneg.so" "returnneg" "int")
returnneg@7F1E4B1D2645
> (returnneg)
4294967295


After having a look at newlisp source code in nl-import.c, I realize that
libffi is not used if str-param-type is empty (nl-import.c at lines
148-149 for Unix platforms). So, I have to cheat a bit and make my call
this way to get the expected result:

Code: Select all
> (import "./returnneg.so" "returnneg" "int" "void")
returnneg@7F1E4B1D2645
> (returnneg)
-1
jef
 
Posts: 11
Joined: Sun May 18, 2014 8:42 am

Re: I/O multiplexing on pipes

Postby Astrobe » Mon Jun 02, 2014 12:05 pm

int returnneg(void) is a valid prototype.It's actually "more correct" than int returnneg(): the former tells that the function takes no arguments, while the latter can take any number of arguments.

source: http://www.drdobbs.com/c-and-c-case-stu ... /184401562
Astrobe
 
Posts: 39
Joined: Mon Jan 11, 2010 9:41 pm

Re: I/O multiplexing on pipes

Postby ralph.ronnquist » Sun Jun 15, 2014 2:30 am

How about net-select?
http://www.newlisp.org/downloads/newlisp_manual.html#net-select
It's documented to deal with sockets, but at least on linux this means file descriptors in general (implemented at nl-sock.c#p_netSelect).
Code: Select all
(net-select (list fd1 fd2) "read" -1)

This doesn't beat the pleasure of attaching C code of your own, but it does seem to offer the functionality.
ralph.ronnquist
 
Posts: 213
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: I/O multiplexing on pipes

Postby jef » Sun Jun 15, 2014 5:52 pm

ralph.ronnquist wrote:How about net-select?.


Yes Ralph, you're totally right, it's exactly what I was looking for.
Works perfectly.
jef
 
Posts: 11
Joined: Sun May 18, 2014 8:42 am


Return to newLISP in the real world

Who is online

Users browsing this forum: No registered users and 2 guests

cron