Remote folder synchronization with NewLISP

[Fritz]

I`m going to create a small program, which will synchronize contents of two folders via TCP-IP.

Now I have an idea to start two newLISP servers and communicate via (net-eval) command. Something like

(net-eval "192.168.1.94" 4711 '(write-file "test.txt" (get-file "http://192.168.1.7/test.txt")))

But this method seems to be unsafe: any villain can make my server to evaluate something evil.

May be there is some other way to send remote commands?

[Lutz]

You could set up a SSH tunnel for 'net-eval' (only on UNIX like OSs):

Code: Select all

ssh -L 1234:remote.com:9999 joe@remote.com

you will be asked for the password of user joe on remote.com and a ssh shell for the remote machine opens. Now start the newLISP server on remote.com and leave the shell open:

Code: Select all

newlisp -c -d 9999 &

Now on you local machine do:

Code: Select all

(net-eval "localhost" 1234 '(+ 3 4)) => 7
; or do
(net-eval "localhost" 1234 "(+ 3 4)") => 7

All traffic will go encrypted through localhost:1234 to remote.com:9999

[Fritz]

You could set up a SSH tunnel for 'net-eval' (only on UNIX like OSs)

Yep, one of PC's has the Windows OS. But I think SSH will work on Windows too.

[Lutz]

Yes, you can tunnel through SSH on Windows too using Putty or a similar tool. But I am just finding out that in the example, I gave, you still can talk to to the newlisp server malicously if you know the remote port, it is listening on, and you can talk to it unencrypted. So the only thing the example does is encrypting traffic, but not protecting against malicous users talking to the server

I am not familiar enough with SSH, perhaps there is a possibility to tunnel in a more secure form.

[Lutz]

In the example the the remote host is listening on port 9999 for local connections, delivered decrypted by SSH. So if you can limit the remote node to listen only locally and not accept port 9999 connections from remote, then everything is secure. Perhaps with some firewall settimg this is possible.

ps: of course port numbers in the example are deliberate and can be changed. Note, that on UIX systems you need super user privileges to listen on port numbers < 1024.

[Fritz]

May be it will be more safer then to transfer data via http, .cgi and POST.

[Lutz]

HTTP mode of read/write/append/delete file operations is already built in to newLISP. In the functions 'read-file','write-file','append-file' and 'delete-file', you can specify a URL for the file name. This works for both, Apache and newLISP server. Internally all file operations are then translated into GET, PUT and DELETE requests.

Run newLISP server in http-only mode to avoid it being abused with 'net-eval':

Code: Select all

newlisp -http -d 1234

[Fritz]

I failed to expain Apache2 "Script PUT" directive. So I have solved it this way (on pure newLISP):

upload.cgi:

Code: Select all

#!/usr/bin/newlisp
#
# Recieves and stores remote file
# 
(module "cgi.lsp")
(module "crypto.lsp")

(set 'upload-dir "tmp/")

(print "Content-Type: text/html\n\n")

(if-not (= (CGI:get "password") (crypto:md5 "password"))
  (begin (print "Access denied.") (exit)))

(set 'flnm (append upload-dir (CGI:get "filename")))

(write-file flnm (hex-unpack (CGI:get "data")))

(if (= (crypto:md5 (read-file flnm)) (CGI:get "md5"))
  (print "Success.")
  (print "Fail."))

(exit)

Remote code:

Code: Select all

; Usage: (put-remote "/tmp/" "test.doc")
(define (put-remote pth flnm)
  (post-url "http://localhost/cgi-bin/upload.cgi" (append
    "password=" (crypto:md5 "password") "&"
    "filename=" flnm "&"
    "data=" (hex-pack (read-file (append pth flnm))) "&"
    "md5=" (crypto:md5 (read-file (append pth flnm))))))

; Sample file transfer
(put-remote "/tmp/" "fridl.7z")

Hex packing functions:

Code: Select all

; Usage: (hex-pack (read-file "test.7z")) => "377ABCAF271C0002292C1..."
(define (hex-pack linea)
  (join (map (fn (x) (format "%02X" x)) (unpack (dup "b" (length linea)) linea))))

; Usage: (write-file "test.7z" (hex-unpack "377ABCAF271C0002292C1..."))
(define (hex-unpack linea)
  (join (map (fn (x) (pack "c" (int (append "0x" x)))) (explode linea 2))))

This combination works. My hex-packing functions are extremely slow but on files below 10Mb it is not important.

[Lutz]

Code: Select all

My hex-packing functions are extremely slow but on files below 10Mb it is not important

You could use 'base64-enc' and 'base64-dec' instead, they are fast.

[Fritz]

You could use 'base64-enc' and 'base64-dec' instead, they are fast.

They are fast, but they use "+" in some circumstances (russian letters):

Code: Select all

(base64-enc " \" remote-file?: Нет ответа о")
"ICIgcmVtb3RlLWZpbGU/OiDQndC10YIg0L7RgtCy0LXRgtCwINC+"

This "+" is not good for POST-URL.

Update: solved this with replace

Code: Select all

(define (base62-enc text)
  (replace "=" (replace "+" (replace "/" (base64-enc text) "_s") "_p") "_e"))

(define (base62-dec text)
  (base64-dec (replace "_e" (replace "_p" (replace "_s" text "/") "+") "=")))

It is twice as slow as base64 ones, but still ten times faster, than my home Internet connection.