Opening Win32 pipe strange behaviour

Machine-specific discussion
Unix, Linux, OS X, OS/2, Windows, ..?
pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Opening Win32 pipe strange behaviour

Post by pjot »

Hi Lutz,

Long time no speak! In the meantime I have been using newLisp for a couple of programs. I discovered a very strange thing however.

With my GTK-server it is possible now to use Win32 pipes. From the client script point of view, these pipes behave like a file. So in order to communicate, the newLisp script has to open a named pipe for writing, and also another named pipe for reading; this is how I have implemented it in the GTK-server.

With newLisp as client, I have programmed this as follows:

(set 'pipeout (open "\\\\.\\pipe\\out" "write"))
(println pipeout)
(sleep 500)
(set 'pipein (open "\\\\.\\pipe\\in" "read"))
(println pipein)
(sleep 500)

Now, the strange thing is: the opening as "write" works OK, I receive an integer number as a file handle. But the opening as "read" fails! The 'pipein' variable receives a NIL.

After some testing, I found out that the problem is the "read". Since if I try to open the 'out'-pipe as "read", I get the same problem.

I would like to mention that opening pipes on Linux work OK. Also other type of programming languages work OK with "read" and "write" on the Win32 pipes.

What am I doing wrong with newLisp in Windows?

Regards

Peter

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Nn Linux and other UNIXs the opening/reading and writing of pipes is done with the same functions as for files and named pipes are created using a OS system level call: mkfifo, This is not the case in Win32 where a completely different API is used to create/open/read/write named pipes.

This is the reason that named pipes are only workng on the Linux/UNIX versions of newLISP and will never be implemented on the Win32 version of newLISP.

But you still could use the newLISP 'pipe' functions, but it is not for named pipes, but else a pipe mechanism working like a named pipe.

Another possibility is to import the Win32 functions necessary with the newLISP 'import' function and write a 'namepipe.lsp' module similar to odbc.lsp etc., but I do not have the time for this at the moment.

The Win32 API's on the system level are so much different from the APIs used on the UNIX family of operating systems. And it is just too much work to deliever this in a transparent manner in newLISP, so that these difference are not visible from a user perspective.

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Hi Lutz,

Thanks for your reply.

Still, I find it very peculiar that with newLisp an "Open for Writing" seems to work fine with the Windows Named Pipes. Only the "Open for Reading" is malfunctioning!

You say that Named Pipes on Windows never will be implemented; I can understand that for the server-side of Named Pipes. Indeed a Named Pipe server on Windows is very different compared to Linux/Unix. The Win32 API for Named Pipes is rather clumsy, if you ask me; I experienced many problems implementing a Named Pipe server in my GTK-server.

But for the client-side access it should be really easy, since a Named Pipe behave like a file. As client, you only have to open this file. The Windows-function for opening files/directorys/disks/pipes/mailslots is in all cases the same "CreateFile" Win32 function. Hence my question: why does opening an existing Named Pipe for "write" works OK with newLisp, and why does opening of an existing Named Pipe for "read" fail?

Other languages like Scriptbasic, VB script, Yabasic etc. can open the GTK-server Named Pipe for "read", so the server-side implemented in the GTK-server appear to work well...?

Regards

Peter

(Actually I am testing on WinXP now.)

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Actually named pipe seems to work perfectly on Win32, at least for me running WinXP home edition. But you have to open for write and read on different programs or newLISP invocations:

In first Command Prompt:

C:\newlisp>newlisp
newLISP v8.0.9 Copyright (c) 2004 Lutz Mueller. All rights reserved.

> (open "ptest" "w")
3
> (write-line "hello" 3)
"hello"
>

and in second Command Prompt:


C:\newlisp>newlisp
newLISP v8.0.9 Copyright (c) 2004 Lutz Mueller. All rights reserved.

> (open "ptest" "r")
3
> (read-line 3)
"hello"
>

What you can not do In Windows is pipe in and out of a named pipe from the same application.

Lutz

also: when running the same newlisp.exe in a cygwin shell then you can open for both read and write as in Linux, I guess in my example above I am really just sharing a file. Look in the file nl-filesys.c for the function:

int openFile(char * fileName, char * accessMode, char * option)

which is opening files in newLISP

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Here is the trick to make it working opening both, read/write, in the same application of newLISP under Win32 using the 'non-blocking' option:

C:\newlisp>newlisp
newLISP v8.0.9 Copyright (c) 2004 Lutz Mueller. All rights reserved.

> (open "ptest" "w")
3
> (open "ptest" "r" "n") ;; << use the non-blocking option!
4
> (write-line "hi there" 3)
"hi there"
> (read-line 4)
"hi there"
>

Lutz

its also mentioned in the manual for 'open'

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Hi Lutz,

Thanks for your replys!

Still it does not work, but I see you use newLisp 8.0.9, while I am using 8.0.6. Maybe this makes a difference? I will upgrade this evening.

Actually, I do not want R/W access to the same pipe. The GTK-server creates 2 independent, individual pipes, a Named Pipe called "in" and a Named Pipe called "out". I have intentionally implemented it this way, since many program languages cannot open a file for R/W at the same time.

So, one pipe must be opened "r" and the other with "w" within the same script. The client script first has to write something to pipe "out", after which the GTK-server puts something to pipe "in".

This last pipe must be opened in "read" mode by the client script, and this is where the trouble starts. If I do this:

(set 'inpi (open "\\\\.\\pipe\\in" "r" "n"))
(sleep 500)
(println inpi)

I still receive a NIL. Again, this is an existing pipe, created by the GTK-server. Also first opening this pipe in "w" mode, after that as "r" "n" delivers a nil. I'll come back with testresults for newLisp 8.0.9.

Regards

Peter
Last edited by pjot on Wed Jul 14, 2004 7:52 pm, edited 1 time in total.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

8.0.9 should be the same as 8.0.6 if it comes to opening file/pipes etc. perhaps you want to have a look in tot the 'C' function I am using. As you did the pipe stuff for GTK-Server in 'C', perhaps you spot something regarding the flags I am using:

Code: Select all

 open( fileName, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE | blocking ) );
I just realize the non-blocking option is commented out for win32 and further testing shows me that opening for read/write from the same invocation of newlisp works (most) of the time. So on win32 the variable 'blocking' is always 0.

I wonder what kind of flags you use when opening/creating pipes.

Lutz

ps: instead of:

(set 'inpi (open "\\\\.\\pipe\\in" "r" "n"))

you could do:

(set 'inpi (open {\\.\pipe\in} "r" "n"))

saving some quotes

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Well, in my GTK-server I create a Named Pipe for Win32 as follows (API only works on NT/2000/XP):

hPipe2 = CreateNamedPipe(
"\\\\.\\pipe\\in", // pipe name
PIPE_ACCESS_OUTBOUND, // write access
PIPE_TYPE_MESSAGE | // message type pipe
PIPE_READMODE_MESSAGE | // message-read mode
PIPE_WAIT, // blocking mode
PIPE_UNLIMITED_INSTANCES, // max. instances
MAX_LEN, // output buffer size
MAX_LEN, // input buffer size
NMPWAIT_USE_DEFAULT_WAIT, // client time-out
NULL); // no security attribute


(As you can see, the name of the pipe is hardcoded here, but this is intentionally, since the client script will create a file handle for the pipename anyway.)

As I can see from your source, you use the generic C 'open' function for READ_ONLY access like this (nl-filesys.c line#416):

return(open(fileName, O_RDONLY | O_BINARY | blocking, 0));

As you mention, the variable 'blocking' has the value 0 here, so the relevant flags are O_RDONLY and O_BINARY. This seems perfectly OK. Compared with your open for "write" (line #420):

return(open( fileName, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE | blocking ) );

the O_CREAT and O_TRUNC flags are extra, but these concern the creation and truncation of the new file. The Linux manpages mention explicitely that if the file is a FIFO, the O_TRUNC will be ignored. Also it is mentioned that the O_NONBLOCK option only will have effect when using FIFO's!

In other words: it might help that the O_NONBLOCK actually will be used in your sourcecode for Win32. Why did you ever comment it out?

On my side, maybe it helps to create the Named Pipe with R/W access, instead of using "OUTBOUND" only. I am going to test that now.

And thanks for the {} remark! The code will be more readable indeed.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

The Win32 version would not compile with O_NONBLOCK. This constant/option exists only in Linux/UNIX header files, no way to use it on Win32 :-(

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

I think I have found the problem.

Look at this testprogram:

--------------------------------------

#include <stdio.h>
#include <fcntl.h>
#include <sys/stat.h>

int main() {

int n;

printf("Connecting to PIPE 'out'...\n");
if ((n = open ("\\\\.\\pipe\\out", O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE)) < 0){
printf("Open FAILED!\n");
exit(-1);
}
else printf("Filehandle is: %d\n", n);

printf("Connecting to PIPE 'in'...\n");
if ((n = open ("\\\\.\\pipe\\in", O_RDONLY | O_BINARY, 0)) < 0){
printf("Open FAILED!\n");
exit(-1);
}
else printf("Filehandle is: %d\n", n);

printf("All pipes can connect!\n");

return 0;
}

--------------------------------------

You see I use your 'open' constructs to test. When I compile with MinGW 3.1, and test it with my GTK-server, this is the output:

C:\Gtk-server\Gtk1>pipe
Connecting to PIPE 'out'...
Filehandle is: 3
Connecting to PIPE 'in'...
Filehandle is: 4
All pipes can connect!

So this looks good. But now, when I compile with Borland 5.5, this is the output:

C:\Gtk-server\Gtk1>pipe
Connecting to PIPE 'out'...
Filehandle is: 3
Connecting to PIPE 'in'...
Open FAILED!

It seems a problem of the Borland compiler. I saw 2 patches for these Free Commandline Tools 5.5, which I will apply and test now.

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Hmmm... I applied the 1st patch, recompiled the testprogram, same result... then the 2nd patch, recompiled the testprogram, also same result.

Actually, my GTK-server for Windows itself is compiled using MinGW since I need to do some GTK stuff, and the GTK port for Windows is optimized for MinGW.

I guess we face a compiler incompatibility here?? Though the generic 'open' should work regardless the compiler.

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Pjot.. Seems you have your BUG day today... First this Ksh 'alias' memory overflowbug under unix and know this Borland issue... whazzzup ;-)...
-- (define? (Cornflakes))

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Yeah... that was big fun!

-----
alias ls='ls'
for i in `ls`; do echo $ i; done
-----

KSH Stack Overflow!

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Actually I (adamss3 pointed me to it) tried to go with MinGW, expecting it to be more compatible with GCC on Linux, but never could get it compiled, too much stuff missing/different. Perhaps I will try again, but it will take some time. These ports look easy at first, but require tons of testing for subtle differences.

Until then: is it not possible to connect to GTK-Server using normal pipes?

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Well, in Win32 the TCP interface works fine. Regarding the pipes: I guess I have to use Named Pipes so the client script knows which "files" it has to open. Nameless pipes might work, but then the client script has to be started by the GTK-server. This will deliver some unconvenient usage complications.

Anyway, I myself used Borland many times in the past, but I have found compilation with MinGW more convenient; the source compatibility is much bigger, and also, the resulting binary is smaller when using MinGW. The test program above for example, has a size of 16106 bytes when compiled with Mingw, but a size of 54272 when compiled with Borland; that is more than 3 times bigger.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Are there any runtimes libraries to install (like on the CYGWIN version of newLISP) ?

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

No extra DLL's needed. The nice thing of MinGW is, that you do not an external DLL to run your program, as is the case with Cygwin. Actually, this was the reason why the MinGW project was started. The MinGW compiler tries to link with msvcrt.dll. Therefore, MinGW compiled binaries can run standalone without any additional libraries.

You can find the MinGW package at http://www.mingw.org

The only thing needed to install is the MinGW-3.1.0-1.exe package. This package contains all include files and object files.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

what about the msys, gcc, binutils, make packages. Are they really not required?

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Yeah you are right, the MSYS package is needed to simulate a SHELL environment.

The other packages are not necessary; they are development packages which can be installed separately.

If you install the MinGW package and the MSYS package, this *should* be enough to make a compilation. Unless you insist on compiling with GCC3.4.0??

First install MinGW, after that the MSYS shell. Then you're all set.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

thanks, started the first compile, looks much better this time around

lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

I've tested the following with Borland:

----------------------------------------
#include <windows.h>
#include <stdio.h>
#include <process.h>
#include <io.h>
#include <fcntl.h>
#include <sys/stat.h>

int main() {

int n;
HANDLE hPipe;

printf("Connecting to PIPE 'out'...\n");
if ((n = open ("\\\\.\\pipe\\out", O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, S_IREAD | S_IWRITE)) < 0){
printf("Open FAILED!\n");
exit(-1);
}
else printf("Filehandle is: %d\n", n);

printf("Connecting to PIPE 'in'...\n");
hPipe = CreateFile(
"\\\\.\\pipe\\in",
GENERIC_READ,
0,
NULL,
OPEN_EXISTING,
0,
NULL);

if (hPipe == INVALID_HANDLE_VALUE){
printf("Open FAILED!\n");
exit(-1);
}
else printf("Filehandle is: %d\n", (int)hPipe);

printf("All pipes can connect!\n");

return 0;
}
----------------------------------------

So for the READ_ONLY part, I use the CreateFile function. When compiling with Borland, this works OK; I can connect to the Named Pipe of the GTK-server:

C:\Gtk-server\Gtk1>pipe
Connecting to PIPE 'out'...
Filehandle is: 3
Connecting to PIPE 'in'...
Filehandle is: 1996
All pipes can connect!

If you use the __BORLAND__ macro anyway, maybe it is an idea to change only the "open" "r" part using the CreateFile function surrounded by the __BORLAND__ macro? The first argument of CreateFile points to the file/directory/mailslot/pipe.

I guess this will save you a lot of time instead of testing and recompiling the whole newLisp with MinGW.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Thanks for investigating this. I might go MinGW anyway, don't know yet. In the long run MinGW is probably the better solution as all it's tools are GNU based and the same tools used in the Linux/UNIX versions, they will probably converge more and more, while with a proprietary product like Borland, you don't even know how long it will exists. I have some 'emotional' bonds to that BCC compiler, because I worked there for a while (Quatro Pro 6.0, non-US international versions).

If the MinGW port gets too tough I will go with the Win32 CreateFile() solution, and try again 1/2 year later with MinGW. They seem to get closer and closer to be 100% compatible with GNU on Linux/UNIX.

Lutz

adamss3
Posts: 14
Joined: Mon Oct 14, 2002 11:53 am

Post by adamss3 »

I'd vote for staying with the Borland compiler for ease of changes. Last time, therre were a number of changes that would need to be done to move to MinGW.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Turns out that the latest MinGW-3.1.0-1 is pretty close to Borland BCC in required changes. With little differences it can use the same #ifdef's. The only mayor thing was the return value of vsnprintf() and the naming of FILE structure members.

I have posted a newlisp-mingw.exe in the http://newlisp.org/downloads/development directory perhaps Pjot can tell us if it works with named pipes on the GTK-Server. I did not much test it, so expect everything :)

newlisp-mingw.exe is about 25% bigger than the executable generated with Borland BCC and I could reduce this a tiny bit by changing the optimization options.

For now I will stay with BCC for the binary Win32 distribution but include a new makefile_mingw in the source distribution for people who want to compile with MinGW, whch definetly has advanced quite a bit during the last year or so and is an OSS alternative to Borland BCC.

Still, I think that CYGWIN does the best Win32 solution if it comes to source code compatibility with Linux. This is because they can hide all differences in the addtional libraries, which have to be installed. Borland and MinGW depend solely on the runtime libraries installed by Wndows.

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

That's cool! I'll try it immediately, and let you know the results. Thanx!!

Strange that the resulting binary is bigger than the Borland version. In almost all my programs I experience the other way around...

Locked