development version newLISP v.8.5.9

[Lutz]

Yes the 'perl-clone' or the cygwin fork() are both Win32 implementations of the UNIX fork() by using Win32 threads and code to replicate/clone the parents memory.

I went to the source code of both Perl and Cygwin, its not a small task to understand it and rip it out, usable for newLISP. Anybody who wants to help out here is welcome to work on this, it requires both, good Win32 internals knowledge and UNIX knowledge and it has to be done in a way that thread-process communications by means of shared memory and syncronized by semaphores still work well, open file handles have to be duped etc.

Everything has to be small enough not to change newLISP's small footprint on disk and in memory, which is one of the reasons people like using newLISP.

Lutz

[pjot]

I am just wondering if there was a particular reason why there is no 'fork' emulated in newLisp for Windows. Of course I am willing to help you out but my Win32 knowledge unfortunately is limited... :-(

If it appears too difficult, too much work, or if the resulting newLisp binary becomes too large, well, then probably it might not be a wise thing to do.

Peter

[Lutz]

It is a bigger piece of work, thats why I have not come around doing it. I also believe that most people wanting to do a multithreaded application in newLISP will be found not on Windows but in the UNIX world. So for the Win32 flavor 'fork' doesn't have the same priority as it has on UNIX.

Lutz

[pjot]

Is there maybe a possibility to write a 'fork' emulation with newLisp? The commands 'cpymem', 'process' and so on could be used for it, I suppose?

The whole thing can be put in a separate context, so your newLisp binary will not grow larger.

Still it is an emulation, but it's better than nothing.

Peter

(PS your suggestion about those Unix people I agree with. The whole 'fork' idea for Windows is not that urgent.)

[Lutz]

Yes, you could do that theoretically, interesting idea, although it would be slow.

Lutz

[pjot]

Well, I got something working.

http://www.turtle.dds.nl/newlisp/winfork.lsp


The original "prodcons.lsp" works in Windows now with some small modifications:

1) Removed the (wait-pid) function, does not exist in newLisp for Windows
2) Load the "winfork.lsp" context first
3) Syntax is: (winfork '(function)) , so calling the symbol of the function instead of the function itself.

The "prodcons.lsp" demo program for Windows can be found here:

http://www.turtle.dds.nl/newlisp/prodcons.lsp


Anybody who can improve the WINFORK please do so (and let me know...;-) )

Peter

[Lutz]

Excellent! but here 2 small suggestions to make the code even shorter and faster: (1) use 'source' to convert symbols to their source code:

Code: Select all

(set 'a 123)
(set 'b 456)

(define (foo x y) (+ x y))

(set 'environment "")
(dolist (s '(a b foo)) (write-buffer environment (source s)))

; environment now contains all the source for 'a 'b and 'foo

'source' takes care of all the conversion

(2) when using (write-buffer ...) with a string instead of a file handle, you have a very fast in-memory string append.

Lutz

[pjot]

Nobody can beat the master! ;-)

The WINFORK context is much smaller now and works pretty cool. I uploaded new a version at

http://www.turtle.dds.nl/newlisp/winfork.lsp

Now the 'fork' command can be used in Windows as well!

Peter

[pjot]

Further testing seem to reveal a memory leak in the Windows-version of 'share'.

If you change the 'prodcons.lsp' example to let it start with '(run 100000)' and ALSO comment all semaphore stuff; watch your processlist, the spawned newLisp processes will grow until they stop (at 134mB on my WinXP system).

However, if I change the 'prodcons.lsp' example like this:

Code: Select all

(load "winfork.lsp")

(constant 'wait -1 'sig 1 'release 0)

(define (consumer n)
	(set 'i 0)
	(while (< i n)
		#(semaphore cons-sem wait)
		#(println (set 'i (share data)) " <-")
		(println "Consumer")
		#(semaphore prod-sem sig)
		)
	(exit))
		
(define (producer n)
	(for (i 1 n)
		#(semaphore prod-sem wait)
		#(println "-> " (share data i))
		(println "Producer")
		#(semaphore cons-sem sig)
		)   
	(exit))

(define (run n)
	(set 'data (share))	
	(share data 0)

	(set 'prod-sem (semaphore)) ; get semaphores
	(set 'cons-sem (semaphore))

	(set 'prod-pid (winfork '(producer n))) ; start threads
	(set 'cons-pid (winfork '(consumer n)))

	(semaphore prod-sem sig) ; get producer started

	(sleep 10000) ; make sure semaphores are not released too early
	
	(semaphore cons-sem release) ; release semaphores
	(semaphore prod-sem release)
	)


(run 100000)

(exit)

...then the memory usage is not growing. So it is definitively in the (share).

If I run the original 'prodcons.lsp' in Linux, then the memory usage is OK.

Peter

[pjot]

If I run this on a Win2000 machine the memory usage also increases to 134 Mb (but newlisp does not crash).

[pjot]

Probably the issue is caused by these lines in win32-util.c (lines 224-227):

Code: Select all

UINT * winMapView(UINT hMemory, int size)
{
return((UINT*)MapViewOfFile((HANDLE)hMemory, FILE_MAP_WRITE, 0, 0, size));
}

Every time the function MapViewOfFile is called, it will cost you some additional memory. Therefore each process should only call it once, and then refer to the same returned address every subsequent time.

I also found this remark:

Note also that MapViewOfFile() increments a usage counter that is decremented by UnmapViewOfFile() or on process termination. The physical named object is not removed from memory until the last process has unmapped the object, and the usage counter is set to zero.

...at http://www.jps.at/dev/kurs/3-19.html.

Peter

[Lutz]

Thanks Peter this is fixed for 8.6.0.

Lutz