threadring in newlisp

[Lutz]

Additionally to the pipes created by the threadring program, 'spawn' creates a socket pair (based on Unix local domain sockets) for each process spawned and to be used by the message API with 'send' and 'receive'.

I will make the the creation of these sockets optional in the 'spawn' call. This will allow applications to not use the message API or only use it for slected childs spawned:

(spawn <sym-variable> <child-process> [true])

The optional 'true' parameter would cause the socket pair to be created, while without it, no socket pair would be created for that child.

[Lutz]

See http://www.newlisp.org/downloads/development/inprogress for these changes now implemented.

The threadring program will now run unchanged with a higher number of child processes in the ring.

The negative timing numbers will be addressed at a later time.

[TedWalther]

The socketpair is how the child process returns its final value to the parent, right? What happens if you spawn a process with the socketpair absent? Isn't that the same as a fork?

[Lutz]

The final return value of the child process is transferred via shared memory, not via sockets. Perhaps in the future the low level 'fork' and 'wait-pid' should be taken out in favor of the more high-level and practical 'spawn' plus the message API.

[TedWalther]

Or at least, don't take fork and wait-pid out entirely; put them in the posix module?

I just tested, and the spawn version is 2% slower than the fork version. So, they are almost equivalent.

[TedWalther]

After meditating a bit, about being able to "send" and "receive" between spawned processes, I had this idea. Could you expose the send/receive mechanism just a little bit? That way I could "send" a message to the parent, requestion the communications port/socket/handle for some other child. Then I could communicate directly with that other child. And I could instruct the parent how to handle such a request from the child. That would save having to make a socket pair for every single child/child pair.

Have you found that mmap is faster or slower than SysV shm api? Any thoughts on switching to the pthreads api to double the speed?

[Lutz]

I found the shm API to be the slowest followed by memory sharing with semaphores, followed by pipes and socket pairs beeing the fastest. I implemented all of these completely and compared speed on different platforms.

The pthreads API happens on only one core on most platforms and in the same process space, and because of that, doesn't scale well on multicore CPUs. Look at cores-used stats in the Debian benchmarks. Also, when running in the same process space, newLISP internal memory structures get messed up, if not duplicated for each thread.

Regarding, child to child communications, for now I suggest using either the proxy approach as outlined in the docs, or creating pipes, which works almost as fast as socket pairs. I understand that the best would be, if the same 'send' and 'receive' could be used, but haven't found yet a viable solution. Ideas, how to do this abound, but the devil lies in the details. Any solution needs some central registry for resources, either maintained by the OS kernel or by the newLISP parent process.

At the moment, the most important thing is, to use the existing API and find typical use patterns for solving real world programming problems. From there we will know, how important it is to extend send/receive by child to child comm., or how much can be done with the existing API.

On the Tips & Tricks page, there is a new category "Parallel and Distributed Processing" with an example, how to do parallel web page retrieval. We need more real world examples to show using the 'spawn' API.

[Kirill]

On the Tips & Tricks page, there is a new category "Parallel and Distributed Processing" with an example, how to do parallel web page retrieval. We need more real world examples to show using the 'spawn' API.

My YubiKey verification library uses (spawn) to query verification servers in parallel. It's quite basic, but real world anyway.

[Lutz]

Thanks for the hint. Parallel network access is definitely a good candidate for 'spawn', as is anything else waiting for I/O.

[cormullion]

My only use of spawn is in my irc-bot:

Code: Select all

(spawn 'p1 (eval-string tmp))
(if (sync 2000)
    ; looking good
    (set 'result (string p1))
    ; no luck
    (begin 
        (set 'result (string "error:" p1))
        (abort)))

[Lutz]

What is it, you have in tmp ?

[cormullion]

Usually just any newLISP string: e.g. on #newlisp at the moment

cormullion: (+ 2 3)
newlithp: 5

still a work in progress, though :)