Problem with UDP

Q&A's, tips, howto's
Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Actually it was not much work, I just had to split net-send-udp and net-receive-udp in versions with and without creating and closing the socket. What took time was testing it and writing the documentation :-)

lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hi Lutz,

Just out of curriosity :), why did you chose to create 2 new functions
and not create more options with existing ones?

Its just a slight thought that (net-send-to) and (net-send-udp) could have
been mixed (because net-send-udp creates a real seperation in the newlisp network functions and is very clear to the user, while i think it could confuse the user now there are 2 ways to goto Rome ;-), while just the differance in variables in the function would create the differance in behaviour...


i.e --->

(net-send-udp str-remotehost int-remoteport str-buffer [bool])
and
(net-send-to str-remotehost int-remoteport str-buffer int-socket)

only differ [bool] or int-socket so it could become -->

(net-send-udp str-remotehost int-remoteport str-buffer [int-socket] or [bool] or [ nothing ])



and for --->

(net-receive-udp int-port int-maxsize [int-microsec][str-addr-if])
and
(net-receive-from int-socket int-max-size)

differ the amount of variables and could become -->

(net-receive-udp [ int-socket int-max-size ]
or
[ int-port int-maxsize [int-microsec][str-addr-if] ])


Its just a personal taste but i like the compacked amount
of functions newlisp has and makes it with the enhancement in
the manual more logical?


What do you think?


Regards, Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

I thought about this too and internally there are really only two functions (which take socket creation/close options), but then I realized that sendto() and recvfrom() can also be used for non UDP communications, following code works perfectly:

=== SERVER
> (net-listen 1234)
1984
> (net-accept 1984)
1976
> (net-receive-from 1976 20)
("hello" "0.0.0.0" 0) ;; shows 0.0.0.0 for default adapter
>

=== CLIENT

> (net-connect "127.0.0.1" 1234)
1956
> (net-send-to "127.0.0.1" 1234 "hello" 1956)
5

In fact in this scenario you can mix match net-send-to/net-send and net-receive-from/net-receive to your liking. This is TCP communications not UDP. Note that at the server the sender port comes as a 0, because of TCP protocol this field is not used. Using the new functions might be faster when on small data sizes in TCP (haven't measured it yet). For this reason I thought it was better to show the user two pairs of functions to make the difference visible.

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

Yes it makes good sence and is flexible also but its also double syntaxing
for the use of tcp (the socket and the remote ip .port). Indeed a mix match position...

For tcp most people would use the way of setup a connection and then
send data to that socket/bind.. actualy with the (net-send-to) it has only
doubled the variables..for UDP its more logical because of the protocol.

It could be a fraction quicker using (net-send-to) under tcp but thats
purly internaly in newlisp and your creating a tcp alike udp ;-)

For the conventional way of tcp communication i would always choose the
bind -> sendto <- recvfrom way, because you always want to know who is on the other end of the line.... On Udp where the protocol is not depending
on stability, variations are possible and everyone knows it while using...

You could ofcaurse bring the (net-send-to) and (net-receive-from) under in variables for (net-send) and (net-send-udp) thus keeping tcp and udp seperated... On the other hand a slight enhancement in the Manual that (net-send-to) and (net-receive-from) can be used with tcp too might do the
trick, so that you have 2 real new netowrk functions not just for udp.

Regards, Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Perhaps we need a whole new chapter in the users manual giving templates how to set up client/server code for both protocols.

By now many programmers are familiar with what you can do with TCP, but few know the importance of UDP. I ran into it when somebody asked for UDP to control video equipment and then I implemented net-send/receive-udp.

I wonder how you got involved with UDP, is it network gameing ?

What also would be great to have some 'Norman's 5-cent' stuff for UDP (innocent hint ;-))

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hi Lutz,

Little testing on WINdows with version 8.1.0-Rc1...

Seems to be a problem with (net-receive-from), its not ctaching
anoything from the remote site. Only tested on Win2000 not on linux yet..

Example below, perhpas you see the issue?


**** SERVER

value expected in function net-listen : "127.0.0.1"

> (net-listen 5555 "127.0.0.1" "udp")
104
> (net-send-to "127.0.0.1" 5555 "testing" 104)
7
>



**** CLIENT

> (net-connect "127.0.0.1" 5555 "udp")
292
> (net-receive-from 292 512)
---- waiting and waiting...while server is already read sending...
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

You are not sending to the correct port. The parameters you specify in net-send-to are those for the remote computer. You don't knows its port until your first 'net-receive-from'. Always the client has to initiate the conversation (in this case LINUX):

;=== SERVER
(net-listen 5555 "127.0.0.1" "udp") => 5 ; socket 5
5
(net-receive-from 5 20) => ("hello" "127.0.0.1" 32769)
(net-send-to "127.0.0.1" 32769 "hi there" 5)

;=== Client in a different shell on the same computer 127.0.0.1
(net-connect "127.0.0.1" 5555 "udp") => 6 ; socket 6
(net-send 6 "hello")
;; or (net-send-to "127.0.0.1" 5555 "hello" 6)

(net-receive 6 'buff 20)
;; or (net-receive-from 6 20) => ("hi there" "127.0.0.1" 5555)
buff => "hi there"

The SERVER has to make the first net-receive-from to get the senders port. The senders port, which in this case is 32769 is the port the client uses (picks automatically) when sending to the servers port 5555. Note that on the client side I can use both, net-send-to and net-send and net-receive and net-receive-from. The reason is that the client knows the remote port while the server does not know the remote port until the first net-receive-from.

Lutz

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

You are not sending to the correct port. The parameters you specify in net-send-to are those for the remote computer. You don't knows its port until your first 'net-receive-from'. Always the client has to initiate the conversation (in this case LINUX):

;=== SERVER
(net-listen 5555 "127.0.0.1" "udp") => 5 ; socket 5
5
(net-receive-from 5 20) => ("hello" "127.0.0.1" 32769)
(net-send-to "127.0.0.1" 32769 "hi there" 5)

;=== Client in a different shell on the same computer 127.0.0.1
(net-connect "127.0.0.1" 5555 "udp") => 6 ; socket 6
(net-send 6 "hello")
;; or (net-send-to "127.0.0.1" 5555 "hello" 6)

(net-receive 6 'buff 20)
;; or (net-receive-from 6 20) => ("hi there" "127.0.0.1" 5555)
buff => "hi there"

The SERVER has to make the first net-receive-from to get the senders port. The senders port, which in this case is 32769 is the port the client uses (picks automatically) when sending to the servers port 5555. Note that on the client side I can use both, net-send-to and net-send and net-receive and net-receive-from. The reason is that the client knows the remote port while the server does not know the remote port until the first net-receive-from.

Lutz

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Here is a short example for UDP server/client communications, start the server first:

=== SERVER ===

Code: Select all

#!/usr/bin/newlisp

(set 'socket (net-listen 1001 "" "udp"))
(println "server listening on port " 1001)
(while (not (net-error))
	(set 'msg (net-receive-from socket 255))
	(println "->" msg)
        (net-send-to (nth 1 msg) (nth 2 msg) (upper-case (first msg)) socket))

;; eof
=== CLIENT ===

Code: Select all

#!/usr/bin/newlisp

(set 'socket (net-connect "127.0.0.1" 1001 "udp"))
(while (not (net-error))
	(print "->")
	(net-send socket (read-line))
	(net-receive socket 'buff 255)
	(println "=>" buff))

; eof
Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

>Perhaps we need a whole new chapter in the users manual
> giving templates how to set up client/server code for both protocols.

Good idea...

>I wonder how you got involved with UDP, is it network gameing ?

Well.. no not directly, its a long story but actualy i just happen to be e network junky in languages. Its realy good that newlisp has good UDP support, as not many languages support UDP good or not at all.

>What also would be great to have some 'Norman's 5-cent' stuff for UDP >(innocent hint ;-))

Ooo yes somethings are upcoming, im very bussy this month but have two
nice projects for udp on my list, ill post them when ready... :-)

Thank for the explanation on the net-receive-from, I was out of way :)

Regards, Norman.
-- (define? (Cornflakes))

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

Well amazingly I spend the last two days thinking what is wrong with the
UDP in newlisp communication. Well finaly i grabbed my good old sniffer and
started analizing the packets flying over the copper using my UDP newlisp client that was communicating towards a Remote UDP-server (not newlisp).

My client was unable to communicate, as i figured out after 4 hours thinking
that i was doing something wrong. But its seems my expectations from the
UDP implant where different then expected.

See my client communicates towards a remote UDP-server. This remote
UDP server listens for incoming sessions on a specific port. When data comes
in on that port it will start sending it own data to the connected client (which is the newlisp udp client) And now the trick !


The server listens on port : 1000 UDP

--> The client connect to server:1000 on UDP, source port 32222 desti: 1000
<-- server sends reply to client with source port:8888 desti port: 32222
--> client is listening for data on remote-port source 1000, And here it goes wrong !!

Strickly when communicating on UDP the client server communication
goes trough source and destination frame from UDP based on addressing
from IP frame.

So eventualy when UDP communicates it communicates on source and
destinationport inside the UDP header. And that does not happen in newlisp.
Newlisp expects packets to come from the Serving-port (1000 in this example) as the UDP header contains the real source port newlisp is unable
to receive the data coming from the UDP server. That was my initial problem
i wrote here the first time...

Actualy newlisp should be able, when acting as a client i.e. (net-receive) in udp-mode to check the source port for every incoming packet
from the server and send back on that port with (net-send).
For the server-side its oke because it is sending to the right port but newlisp is currenlty listening on the serving-port instead on the port in the udp header...

I tried to configure it, but im unable to get newlisp to listen on incoming
data from an unknown source port...perhpas you have a hint ?

Well its a RFC issue and I hope you understand my story above...

My question, Are you able to do something about it perphaps ? ;-)

Regards, Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

>>>
Actualy newlisp should be able, when acting as a client i.e. (net-receive) in
udp-mode to check the source port for every incoming packet
>>>

yes, use net-receive-from on the client to find out the port the server used to send.

--> The client connect to server:1000 on UDP, source port 32222 desti: 1000
correct
<-- server sends reply to client with source port:8888 desti port: 32222
correct

--> client is listening for data on remote-port source 1000, And here it goes wrong !!

No, there is not such thing as listening on "remote-port source 1000"

The sender never defines it's senders port, only the taget receiving port.

The client listens on the socket it used to send its first message to the server. The server then targets the port of the client it receives in the udp message. See my client/server example from a previous message.

Perhaps you can give me some 'C' code to see how your clients carries on communivations with a newLISP udp server.

You client should send to the 'well-known' (RFC-lingo) port of the newLISP server. The newLISP server will send back to port it reads in the client's UDP package. The client just listens on the same socket it originally used to send out the first packet.

Lutz

ps: the client should never have the need to use net-receive-from, becuase he knows the remote server listen port, when he (the client) initiated the conversation. The client just keeps listening on the sock first used to contact the serve (see the 'C' code of my next post)
Last edited by Lutz on Thu Aug 05, 2004 12:26 am, edited 1 time in total.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Norman, I made a little 'C' program for you to show how to talk to a newLISP UDP server here: http://www.newlisp.org/udp-client.txt

the newLISP udp-server.lsp is here: http://www.newlisp.org/udp-server.txt

The lisp part is the same as posted in this thread previously together with the newLISP client.

Start the udp-server.lsp first then use the 'C' client:

udp-client "127.0.0.1" 1001

When you enter something at the client it gets echoed as uppercase.

You can compile the udp-client.c in two flavors: using sento() / recvfrom() or using send() / recv(). Both work well.

The difference is that in sendto()/recvfom() address info is part of the data packet in the "stuct sockaddr_in" 'C' structure while using send()/revv() this structure is bound to the socket by connect().

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hi Lutz,

Driving for two hours this morning, having a lot of time to think during traffic jams...i think I have found a trick to workaround my problem.
Ill see if it works. Ill test your C programs too.. Ill get back to you..

Thanks..Norman.
-- (define? (Cornflakes))

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hi lutz,

Oke ill explain it once more with an example,
The server is a NONE newlisp program.
So only focus on the client behaviour :-)


Options #1 communication on UDP -->
In this example (net-receive) on the CLIENT side works fine.

*** server listens on port 54054

--> client UDP packet to server (SRC: 32222 | DST: 54054 )
<-- server UDP packet to client (SRC: 54054 | DST: 32222 )
--> client accepts packet.


Options #2 communication on UDP -->
In this example (net-receive) on the CLIENT does not work.

*** server listens on port 54054

--> client UDP packet to server (SRC: 32222 | DST: 54054 )
<-- server UDP packet to client (SRC: 444444 | DST: 32222 )
--> client waits....waits....waits...waits..

The server initiates its Own session to send a reply on what the
client has send. The differance is the SRC-port from the server.
This port is unknown by the newlisp sockets and thus a (net-receive) or (net-receive-from) or (net-receive-udp) always in a wait-forever...

Looks like net-receive does check on source port but does not dynamicly
change the socket options of that sessions internaly.

I agree with you if you say, that way of server communication is odd.
But it is actualy the same as tcp works.

I hope its a little more clear and probably you will tell me to take a hike
and have a beer somewhere on the hill ;-) Just tell me what you think of it...


PS: actualy i had this morning a workaround on windows for the problem.
And that is to initiate a (net-listen) on the (net-local) of the client after sending the data to the server and befor receiving data from the server.
Not realy charming because its a second sessions but i was able to receive data from the server. And currently im unable to reproduce this under linux...odd..

Regards, Norman
-- (define? (Cornflakes))

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

So indeed the solution is as follows (works on both linux / Windows) ->

(set 'remote (net-connect "remote" 54054 "udp") )
(net-send remote "\000\001\002\003")
(set 'C (net-local remote))
(close remote)
(net-listen C "" "udp")
(net-receive C 'buff 1024)
(net-send C "\005\006\007\008")

etc..voila.. it works..

Question remains. -> why this way?
This is the result of "initial" none-static port communication.

Anyway..workaround available , but not according logic and
not very usefull in client UDP communication, client becomes
server this way ;-)

Regards, Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

You should be able to to do a (net-receive remote maxbytes) without having to create a new socket.

What I suspect is, that your server is closing the socket after sending the last response, which is the 'normal' way to do UDP communications, and the way 'net-send-udp' and 'net-receive-udp' work.

We know that that client 'C' <> newlisp-server non-blocking works and we also know that server 'C' <> newlisp-client works from Peters experiments with GTK-server.

I could make an other example in 'C' for 'C'-server with newlisp-client, but I think that this is redundant as Peter has done it with GTK-server already.

Can you get hold of the server code you are trying to talk to?

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

*** up and running *** :-)
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

After some offline experiments together with Norman on the UDP stuff here a better way, how to code a newLISP UDP client http://www.newlisp.org/udp-comm.txt This will also be included in code examples and the manual.

I also suggest that Peter changes to this method in the GTK-server example, although the old method seemed to work too. (in any case the capability of "udp" in net-connect will stay)

Using (net-listen ... "udp") on both sides guarantees that the local socket is always named (bound to local address:por) correctly.

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Done. I have only tested this in Linux now, later this week I'll have the opportunity to test with Windows.

http://www.gtk-server.org/demo-udp.lsp.txt

Peter

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

BUT: it seems that the original net-receive-udp timeout does not work anymore.

(net-receive-udp 57000 128 50000 "127.0.0.1")

If I use this string in the newLisp prompt, newLisp waits endless. But it should timeout after 50000 microseconds. Using newlisp 8.1.0.

HPW
Posts: 1390
Joined: Thu Sep 26, 2002 9:15 am
Location: Germany
Contact:

Post by HPW »

BUT: it seems that the original net-receive-udp timeout does not work anymore.
I see the same when testing with the newLISP.DLL on windows.
It seems not to get back when the timeout is over.
Instead it makes the timeout when it get's a UDP-message.
Then it makes the pending timeout.
Hans-Peter

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

I am not in the office right now but will check later this evening. There have been some changes in UDP stuff, but 8.10 is identical to 8.1.0-rc2 in this respect.

Lutz

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

I found the problem, the feature got wiped out doing the other UDP additions. I will do a re-release on Friday with version 8.1.1

Lutz

PS: this will not contain the new install_home options, I want to go safe because of changes in newlisp-tk.tcl.

nigelbrown
Posts: 429
Joined: Tue Nov 11, 2003 2:11 am
Location: Brisbane, Australia

Post by nigelbrown »

Lutz,
Would you like to e-mail me the home_install makefiles and newlisp-tk.config.home to try out? I have a few live distros.
Regards
Nigel

Locked