Problem with UDP

Q&A's, tips, howto's
newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Problem with UDP

Post by newdep »

Hello Lutz,

Well i came to the status on where im porting some udp tools to newlisp and im running in to a problem...

The net-send-udp and net-receive-udp are actualy oke but not 100% according to the "protocol" ;-)

What is the case? ->

If an UDP packet is send to the remote it sends with it the source port, the remote server will reply to that source port to be able to communicate.
The newlisp TCP stack is implemented that way, but the UDP stack not.


So what is missing? ->

Actualy the udp Stack needs the same behaviour as the TCP stack, to
be able to communicate towards a server on the current sessions and that
is a complete protocol.

It is a matter of Async communication the same as TCP works. Every UDP
sessions needs to have a (net-sessions-udp) too.

Also every UDP sessions must be able to grab the remote port , that is a
(net-peer 16) for UDP that return the remote ip and port...

I hope you are able to rewrite/change the net-send-udp & net-receive-udp because the current implementation has its limits now.

I think its not a big rewrite, but you know your code better that anyone, I hope you consider the above... :)

the impact would be on:
(net-send-udp)
(net-receive-udp)
(net-sessions) -> (net-sessions-udp)
(net-peer)
(net-local)
(net-peek) "probably wont work on udp, not sure anymore"
(net-select)

If the above it changed you will have a fully UDP async communication way,
the same as TCP works, accept for the network-Timeout in UDP which is buildin into the (net-receive-udp) already OR build be used inside (net-select) !!!

Keep on lisping..
Regards, Norman
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

By definition the UDP (User Datagram Protocol) does not have any sender/receiver handshakig protocol like Tcp does. The send goes into "empty space" and if nobody is listening on the receiving port of the receipient addrress or ip broadcast-space, the packet is lost.

For this reason there is no peek or select for pending bytes to read by UDP. If nobody is listening on the receiving end the packed goes into never-never land.

A UDP packed contains (optional no implemented by all OSs) the port number of the sender. It is available on Windows, Linux and BSD and it could be returned theoretically with the remote ip address on reception. But this port numer is deliberately chosen by the sender. On Linux it seems to be 32768 (hex 8000) most of the time, but on BSD it is jumping around and on Windows counting up starting with some deliberate value. So there is not much sense in returning it to the user, because it might not be available the next moment.

UDP is meant to be a very low level protocol and fast for all of the above reasons. You have to implement any kind of handshaking your self as part of the data you send, or use TCP right away.

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

Yes your completly right on UDP..

Still making client-server independent communication possible there
should be a way for the (net-receive-udp) to know what the source ip & port is.

a (net-peer) on udp could return this. And a (net-receive-udp)
could be provided with a return of the source port?
or perhaps a (net-peer-udp) ?


If the above is possible ,(even net-select would be great here, still a next step)
then the UDP implementation is more complete and able to communicate independent.

Indeed the UDP protocol behavior stays the same but the functions in newlisp
related to UDP would be far more flexible.

I.e.

Currently im unable to setup a UDP listener because im not sure what
the local port was my (net-send-udp) send the packet on, so i cant listen
localy on the port for incoming data. protocols like tftp and Some Game protocols do so..

Regards, Norman
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

The source ip is contained in the list net-receive-udp returns as the second field. When you talk bag to the sender you use a so called "well known" port which is agreed upon prior. Normally this is a pair one listen port for the client and another listen port for the sender.

The sender specifies the address and port of the receiver is does not (cannot) specify the outgoing port, which is picked delieberately by the lower level socket routines. The receiver listens on that port targeted by the sender and knows the ip-address where a packet is coming from. The receiver then sends back a message on the "well known'' agreed upon port where previous sender now is listening on.

Again there is no select, net-receive-udp listens and immedeately returns the UDP packet when it arrives.

A net-peer is not necessary because you know already the address where a packet comes from, from the list returned by net-receive-udp.

The outgoing port used by net-send-udp is picked by the socket implementation of the OS and there is no way to specify those. You can only specify the port you are sending to.

The port targeted by the sender is agreed upon prior in the higher level protocol you are using those ports are kneown as "well known ports" and are listed in the /etc/services table. You can use net-service to query them, i.e.:

(net-service "domain" "udp") = 53

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

Yes we are talking about the same issue..let me go one step further...

if server always send back to the client-source-port then the following
will never work in newlisp for the client to receive the data from the server ->

client - > (net-send-udp "127.0.0.1" 2222 "im going back" )
client - > (net-receive-udp ?????? 512 )
server -> (net-send-udp "127.0.0.1" "remote-source-port" "im going back" )

how can (net-receive-udp) know what its local port was?

Anyway.. i tested in my C code and it works nicely... If the OS does not
provide a source port in de Datagram (because its optional indeed) then
its simply "0" (and the OS needs a fixup ;-)

I hope you think about it with a nice cold beer ;-)

Regards, Norman.
-- (define? (Cornflakes))

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

One night away from holidays... but still hacking. Actually I am implementing the UDP protocol for the GTK-server. I've got it running with the following GNU AWK script:

-------------------
BEGIN{
system("gtk-server udp localhost:50000 &")
# Wait for the server to initialize
pause(1)
# Setup UDP socket to server
GTK = "/inet/udp/0/localhost/50000"
# Now define the GUI
print "gtk_init(NULL, NULL)" |& GTK; GTK |& getline
print "gtk_window_new(0)" |& GTK; GTK |& getline WINDOW
....etc
-------------------

However, with newLisp I have to wait for a UDP packet first, before I can send something to the server (see documentation example with "net-receive-udp"). I wonder why this is the case with newLisp. Why is it not possible to send to a UDP port right away?

If I do this anyway, the "net-receive-udp" command produces a NIL.

Is there a way to get around this?

Peter

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hiya Pjot,

with UDP you can send whatever you want, but the server needs to
be listening to be able to get it..With newlisp you can send! but you cannot
wait for data at the same time, that is because its not async. :-)


PS: its a coincidence that we both work on a UDP problem Lutz ;-)

Norman.
-- (define? (Cornflakes))

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Pjot,

Yes if you are using th Gtk-server to listen and to write on 1 port then your client
on the SAME machine will fail, because you cannot have 2 server on 1 machien running with the same listening port.

Actualy that is my point too (indirectly), see above..

Hope Lutz likes beer ;-)

Regards, Norman.
-- (define? (Cornflakes))

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Actually it is very easy. The GTK-server acts as UDP server, waiting for incoming packets. As soon as the UDP server receives something, it will reply. Then it returns to wait state again. BUT: this all happens on the same port. The port is determined by the client script, so the UDP-server will know to which port it has to send it's packets back.

It is almost like TCP, with the difference that it is UDP... ;-) Again, this is how the AWK demo works, so I know this works... Cheers!

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Ofcaurse it works ;-)

But i think we need to buy a 6-pack for Lutz first :-)
-- (define? (Cornflakes))

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Funny we work on UDP at the same time... let's buy a 6-pack Belgian beer (10%)... ;-)

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

The Belgian would be fine, but the dark one please! ;-)

somehow still I don't get what you guys are up to? If you send a packet with

(net-send-udp receiver-ip receiver-listen-port message)

you have the receiver at receiver port doing:

(net-receive-udp receiver-listen-port max-bytes) => (sender-ip sender-message)


There is no such thing as:

(net-send-udp receiver-ip receiver-listen-port port-where-i-will-listen-for-your-answer message)

sender would have to make port-where-i-will-listen-for-your-answer part of the message itself. Because when you set up your socket for sending you can only specify the target port:

dest_sin.sin_port = htons((u_short)remotePort); /* the target/receiver port */
dest_sin.sin_family = AF_INET;

bytesSent = sendto(sock, buffer, size, NO_FLAGS_SET,
(struct sockaddr *)&dest_sin, sizeof(dest_sin));

Perhaps if you explain me what you are doing in 'C' I can understand?

Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

In my situation, the newLisp script acts as a client, while the GTK-server is the UDP server. So there is an interaction between newLisp and C.

OK here my beta version with UDP (just the mainloop):

------------------------
while (1){
if ((numbytes=recvfrom(sockfd, buf, MAX_LEN - 1, 0, (struct sockaddr *)&their_addr, &addr_len)) > 0) {

/* Terminate incoming string */
buf[numbytes] = '\0';

/* Only determine values when empty */
if (se == NULL) {
se = gethostbyname(inet_ntoa(their_addr.sin_addr));
their_addr.sin_family = AF_INET; /* host byte order */
their_addr.sin_port = htons((long)atol(port)); /* short, network byte order */
their_addr.sin_addr = *((struct in_addr *)se->h_addr);
memset(&(my_addr.sin_zero), '\0', 8); /* zero the rest of the struct */
}

<do_something_with_incoming: buf[], send back pointer: *retstr>

/* Add newline to returned string */
strcat(retstr, "\n");

/* Now send the result back to the socket */
if ((numbytes=sendto(sockfd, retstr, strlen(retstr), 0, (struct sockaddr *)&their_addr, sizeof(struct sockaddr))) == -1) {
Print_Error(2, "Could not send DATAGRAM packet", errno);
}
}
else Print_Error(2, "Could not receive DATAGRAM packet", errno);
}
------------------------

As you can see, I determine the destination port from the port given as argument ('port' is one of the arguments to the GTK-server binary).

I just compiled my beta version with GCC and MinGW, this works. Tested UDP with Rebol in Windows, and the GUI came up.

So the idea is:

loop
client-script: send-udp port
server: receive-udp port
server: send-udp port
client-script: receive-udp port
until callback

The 'port' is in all the lines the same port. With newLisp however, it seems I have to do the following:

client-script: receive-udp port
loop
client-script: send-udp port
server: receive-udp port
server: send-udp port
client-script: receive-udp port
until callback

Now, why do I have to wait for a UDP packet first? Since if I do not, the client script will receive a NIL.

Maybe I am not having the same problem as Norman here; again, I just want to know why my newLisp client script has to wait for a UDP-packet first?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Ok, now I see what is going on:

In your 'C' example the server receives on a aocket and immedeately sends back the replay on the same socket. You

keep the same socket open like in a TCP client/server session where the listener send back on the connected socket.

In newLISP the net-send-udp closes the socket after sending and will not try to receive back on it after response.

Your example:

loop
client-script: send-udp port
server: receive-udp port
server: send-udp port
client-script: receive-udp port
until callback

Actually works with newLISP if both client and server are newLISP. If one of them is GTK-server then

GTK-server tries to send back on the socket wher it received, but that connection is already closed. That is

fundamentally a very different thing.

In newLISP/UDP sockets are immedeately closed on both sides:

net-send-ip:
(1)create socket,
(2)use it to send to receiver address and port,
(3) close socket.

net-receive-udp:
(1)create socket
(2)listen on socket
(3)read udp packet on socket
(4)socket

The reason for this implenation is that UDP was primarily meant to be a protocol with no inherent handshaking, i.e.

when you do a syslog(), you just fire away your log message without waiting for anything back.

Of course you can work it the way you do, but I think than you should do Tcp/Ip. I am not familiar with the Gaming

usage of UDP perhaps the method you are describing is the way is the way they work UDP? The newLISP implementaion

is targeted towards controlling hardware over a network (i.e. video equipment), where the sending of a UDP packet

and it's reception is the complete transaction and anything else happens on a higher level in an application

protocol.

But there may be a solution! The net-connect function already has an undocumented additional "udp" option and I

could add one to net-listen too and then you could work this way:

server:
(net-listen port local-ip "udp") => socket
(while (not (net-select ...) ...)
(net-receive socket)
(net-send socket)

client:
(net-connect port remote-ip "udp") => socket
(net-send socket message)

etc.

So the idea is to work with normal net-send and net-receive but open the sockets for UDP with listen and connect. (the listen would only make the UDP socket not really 'listen' like a stream socket)


Lutz

pjot
Posts: 733
Joined: Thu Feb 26, 2004 10:19 pm
Location: The Hague, The Netherlands
Contact:

Post by pjot »

Hi Lutz,

Good news! Your tip works. Right now, the newLisp script with UDP looks like:

-----------------------
# Define communication function
(define (gtk soc str)
(net-send soc str)
(net-receive soc 'tmp 16)
tmp)

# Start the gtk-server
(process "gtk-server udp localhost:50000")
(sleep 1000)

# Connect to the GTK-server
(set 'socket (net-connect "localhost" 50000 "udp"))

# Define GUI
(set 'tmp (gtk socket "gtk_init(NULL, NULL)"))
(set 'win (gtk socket "gtk_window_new(0)"))
.....etc
-----------------------

This works exactly as expected.

It will take some time before the UDP version of the GTK-server will be released (me being on holidays) but newLisp will be a client language supporting it.

Thank you!

Peter

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

Yes, (net-connect) works that way indeed. What is needed extra
is the (net-listen) on UDP and the solotion is made. :-)

below is exactly what I searched... Now the (net-listen) enhancement would
be great.. Can (net-select) be used this way too ?

> (net-connect "127.0.0.1" 54000 "udp")
300
> (net-local 300)
("127.0.0.1" 1224)

Regards and thanks for the thoughts :-)

Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

net-select, net-peer, net-local and net-peek will not need the udp option, the socket receives the datagram or stream attribute during creation, after that it is just a socket and can be treated with all other functions in a normal way.

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

That is great Lutz, so actualy the adjustment for (net-listen) is left?

Very nice...!
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Yes, I am leaving home now, but you get it tonight.

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hello Lutz,

version linux 8.0.16

Using the (net-send) after opening the udp sessions to the remote (not a newlisp daemon) with a (net-connect sock "ip" "udp") the connection is directly dropped localy...actualy using newlisp <> newlisp (net-send) works fine.. Is the expectation different using none newlisp <> newlisp communication?


Secondly the server site, how to grab the remote peer? -->

> (set 'listen (net-listen 5555 "" "udp"))
5
> (net-peek listen)
0
*** remote client send "hello" with (net-send)
> (net-peek listen)
5
> (net-peer listen)
nil <--- Howto pickup remote peer port in a different way?
> (net-sessions)
()
>

Thirdly.. i think you see it already ;-) the (net-sessions) is not
updated for the "udp" trick...
Not sure if you want to implant this into the core?



Regards,
Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

The missing socket in the 'net-sessions' for 'net-listen' with "udp" option was an oversight and is quick to fix.

The other stuff, I have to look into it a little bit deeper. UDP with 'C' listen(), receive() and connect(), send() is something normally not done. What is used mostly with UDP is using the 'C' sendto() and revvfrom() functions. This is the way 'net-send-udp' and 'net-receive-udp' work, and it looks like getpeername() does not work on SOCK_DGRAM sockets. There is very little about all this found in books and on the internet, I will experiment and find out more about it tomorrow.

Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

No hurries...Take your time...:-)
-- (define? (Cornflakes))

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Hi Lutz,

Yes sendto is often used without the connect function called befor but send
can be used also and can only work when connect is used to open the session
first. I think this is what you already do with the current "udp" trick.. Enhancing the "udp" functionality could take a little rewrite but is generaly
the same as for tcp accept that you need an extra catch on error...

Regards, Norman.
-- (define? (Cornflakes))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Yes, I am just about to upload 8.1.0-rc1 which gets the asynchrounous UDP stuff working.

In UDP mode sockets do not contain the peer information about ip-address and port, they are part of the UDP-packet itself.

I added two new functions:

(net-receive-from socket maxsize) => (message ip-no port)

and

(net-send-to ip-no port message socket)

also (net-receive-udp ...) now returns also the sender port as last member in the list.

This is now how you set up asynchrounous UDP server/client

;===== server

(net-listen port ip-strr "udp") =>sock-no
(while (not (net-select sock-no "r" 100000)) (...))

(net-receive-from sock-no max-bytes) => (message ip-str port)

(net-sent-to ip-str port reply-msg sock-no)

; etc

;====== client

(net-connect server-ip server-port "udp") => sock-no

(net-send sock-no message)
;; or
(net-send-to server-ip server-port message sock-no)

;; poll for reply from server
(while (not (net-select sock-no "r" 100000) (...))

(net-receive sock-no max-bytes)
;; or
(net-receive-from sock-no max-bytes

; etc


Note that on the server 'net-receive-from' and 'net-send-to' are critical because the listen socket has no clue about the remote address.

On the client side both can be used because the connect binds the remote address to the socket.


Lutz

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

You are going like the wind!
And a new release candidate And udp enhancement......:-)



Its already released i see..pfffffffff...
-- (define? (Cornflakes))

Locked