SMTP - Error Trapping and Authentication

Featuring the Dragonfly web framework
Locked
Tim Johnson
Posts: 253
Joined: Thu Oct 07, 2004 7:21 pm
Location: Palmer Alaska USA

SMTP - Error Trapping and Authentication

Post by Tim Johnson »

Hi Folks:

Just tried out the smtp interface for newlisp and it is the easiest that
I've used. It would be helpful however to have information in parsing
the response string to trap errors. If anyone can point me in the right
direction, I would appreciate it. I.E. Should I always expect to find
the response code as the beginning of the response string?

Rebol and python throw errors if a successful transmission is not
accomplished. I can observe that a successful transmission returns
221 at the head of the response string.

In addition - although I can sidestep using any SMTP server that requires
authentication - I think it would be very valuable to be able to send
authentication parameters. I've found one thread in this forum on this
matter, but couldn't find a conclusion. Again, I would welcome comments.

thanks
tim

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

HI Tim. You might be referring to this thread - http://www.alh.net/newlisp/phpbb/viewtopic.php?p=7611. With that code (oh there's a typo I think - send-str shouldn't be quoted in net-send-get-result) I can send and receive email using simple base64 authentication:

Code: Select all

220 smtp.example.com ESMTP Service
AUTH PLAIN aGljeXJpbGhvcGV5b3VyZWhhcHB5bm93
sent: AUTH PLAIN aGljeXJpbGhvcGV5b3VyZWhhcHB5bm93
235 2.0.0 OK Authenticated
HELO example.com 
sent: HELO example.com
250 example.com Hello dyn.example.com [91.122.83.67], pleased to meet you
MAIL FROM: <a>
 sent: MAIL FROM: <a>
250 2.1.0 <a>... Sender ok
RCPT TO: <a> 
sent: RCPT TO: <a>
250 2.1.5 <a>... Recipient ok
DATA 
sent: DATA
354 Enter mail, end with "." on a line by itself
TO: a.user@example.com 
sent: TO: a.user@example.com
FROM: a.user@example.com 
sent: FROM: a.user@example.com
SUBJECT: Greetings 
sent: SUBJECT: Greetings
X-Mailer: newLISP v.9302 sent: X-Mailer: newLISP v.9302
sent:
How are you today? sent: How are you today?
. 
sent: .
250 2.0.0 m2Q96FNw004236 Message accepted for delivery
QUIT sent: QUIT
221 2.0.0 example.com closing connection
true
Perhaps someone with the right knowledge can take up the story from here.

Kirill
Posts: 90
Joined: Wed Oct 31, 2007 1:21 pm

Post by Kirill »

Hello, world!

This is my first post, I've been lurking around here for a while. I'd like to make a note of a small bug in smtp.lsp, that could be squashed if anyone volunteers to add support for SMTP authentication.

Current smtp.lsp doubles single dots on a line by themselves. In SMTP, all dots at the beginning of a line should be doubled (cf. RFC2821 sec. 4.5.2), i.e. not only dots on a line by themselves:
- Before sending a line of mail text, the SMTP client checks the first character of the line. If it is a period, one additional period is inserted at the beginning of the line.
I have a wish to hack a bit on the SMTP module, but have a constant lack of time.

Best regards,
Kirill

P.S. Kirill is Кирилл in Russian, which also can be spelled as Cyril. So we're two Кириллs here. :-)

Tim Johnson
Posts: 253
Joined: Thu Oct 07, 2004 7:21 pm
Location: Palmer Alaska USA

Post by Tim Johnson »

FYI
Regarding my question about error codes, I overlooked something:
send-mail returns nil if the transmission failed.
:-) So never-mind on the error codes.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

The following changed code in smtp.lsp would insert a period in each line starting with a period.

Code: Select all

(define (mail-send-body )
    (net-send-get-result "")
    (dolist (lne (parse mail-body "\r\n"))
        (if (starts-with lne ".")
            (net-sent-get-result (append "." lne))
            (net-send-get-result lne)))
    (net-send-get-result "."))
The function to be replaced is at the end of the file. If somebody can test this, I can include the change in the next release. Unfortunately I don't have a possibility to test this at the moment.

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

Yes, that works OK for me here, if I add the AUTH PLAIN authentication step. (But then the last version did too.)

I'm still puzzled by the difference between my local copy and the official one:

Code: Select all

(define (net-send-get-result str conf)
   (set 'send-str (append str "\r\n"))
   (if debug-flag (println "sent: " send-str)) 
   (net-send socket 'send-str)
   (if conf (confirm-request conf) true))

(define (net-send-get-result str conf)
   (set 'send-str (append str "\r\n"))
   (if debug-flag (println "sent: " send-str)) 
   (net-send socket send-str)
   (if conf (confirm-request conf) true))
The first version is in /usr/share/newlisp/modules/smtp.lsp. How could (net-send socket 'send-str) work? Or have I accidentally modified it...? What's your copy say, Lutz?

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

PS: hello Kirill!

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Up to version 9.3.1 the quirky syntax (net-send socket 'send-str) was allowed but since 9.3.2 the (net-send socket send-str) the quote must be omitted (which most people did anyway).

Tim Johnson
Posts: 253
Joined: Thu Oct 07, 2004 7:21 pm
Location: Palmer Alaska USA

Post by Tim Johnson »

Lutz wrote: If somebody can test this, I can include the change in the next release. Unfortunately I don't have a possibility to test this at the moment.
Besides my localhost, I have accounts with two mail servers that require
authentication. I would be happy to test, but being a newbie I would
need the entire code base with modifications for authentication.

tim

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

Without newlispdoc comments, my version of smtp.lsp is like this:

Code: Select all

(context 'SMTP)

(set 'debug-flag nil)

(define (send-mail mail-from mail-to mail-subject mail-body SMTP-server user-name password)
    (and
        (set 'from-hostname (nth 1 (parse mail-from "@")))
        (set 'socket (net-connect SMTP-server 25))
        (confirm-request "2")
        (unless (and (empty? user-name) (empty? password)) 
           (mail-authorize user-name password))
        (net-send-get-result (append "HELO " from-hostname) "2")
        (net-send-get-result (append "MAIL FROM: <" mail-from ">") "2")
        (net-send-get-result (append "RCPT TO: <" mail-to ">") "2")
        (net-send-get-result "DATA" "3")
        (mail-send-header)
        (mail-send-body)
        (confirm-request "2")
        (net-send-get-result "QUIT" "2")
        (or (net-close socket) true)))

(define (confirm-request conf)
   (and
    (net-receive socket 'recvbuff 256 "\r\n")
    (if debug-flag (println recvbuff) true)
    (starts-with recvbuff conf)))
  
(define (net-send-get-result str conf)
   (set 'send-str (append str "\r\n"))
   (if debug-flag (println "sent: " send-str)) 
   (net-send socket send-str)
   (if conf (confirm-request conf) true))

(define (mail-authorize user-name password)
   (net-send-get-result (append "AUTH PLAIN " (base64-enc (append "\000" user-name "\000" password))) "235"))

(define (mail-send-header)
    (net-send-get-result (append "TO: " mail-to))
    (net-send-get-result (append "FROM: " mail-from))
    (net-send-get-result (append "SUBJECT: " mail-subject))
    (net-send-get-result (append "X-Mailer: newLISP v." (string (nth -2 (sys-info)))))) 

(define (mail-send-body ) 
    (net-send-get-result "") 
    (dolist (lne (parse mail-body "\r\n")) 
        (if (starts-with lne ".") 
            (net-sent-get-result (append "." lne)) 
            (net-send-get-result lne))) 
    (net-send-get-result ".")) 

(define (get-error-text)
    recvbuff)

(context 'MAIN)

; eof
and is tested like this:

Code: Select all

(load "...smtp.lsp")

(set 'SMTP:debug-flag true)

(SMTP:send-mail 
 "from@example.com" 
 "to@example.com" 
 "title" 
 "body" 
 "smtp.example.com" 
 "user.name" 
 "password"))

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Thanks Cormuliion, here is a merged version of yours and the last of the distribution. If it checks out well in testing, we can put it in the next release.

also posted here: http://www.newlisp.org/smtp.lsp
and syntax highlighted: http://www.newlisp.org/syntax.cgi?http: ... g/smtp.lsp

Code: Select all

;; @module smtp.lsp 
;; @description Send mail using SMTP protocol
;; @version 2.0 - 2006-10-08 cormullionx added AUTH PLAIN authentication 
;; @author Lutz Mueller 2001 , Cormullion 2008
;; 
;; <h2>Routines for sending mail</h2> 
;; This module implements routines to communicate with a SMTP mail server 
;; for sending email. To use this module include the following 'load' statement 
;; at the beginning of the program file: 
;; <pre> 
;; (load "/usr/share/modules/newlisp/smtp.lsp") 
;; </pre> 
;; To see debugging information: 
;; <pre>(set 'debug-flag true)</pre> 

(context 'SMTP) 

(set 'debug-flag nil) 

;; @syntax (SMTP:send-mail <str-from> <str-to> <str-subject> <str-message> <str-server>i [<str-usr> str-pass>]])
;; @param <str-from> The email address of the sender.
;; @param <str-to> The email address of the recipient.
;; @param <str-subject> The subject line of the email.
;; @param <str-message> The message part of the email.
;; @param <str-server> The address of the SMTP server.
;; @param <str-user> Optional user name for authentication
;; @param <str-pass> Optional password for authentication
;; @return On success 'true', on failure 'nil'.
;; In case the function fails returning 'nil', the function
;; 'SMTP:get-error-text' can be used to receive the error text.
;;
;; @example 
;;(SMTP:send-mail "jdoe@asite.com" "somebody@isp.com" "Greetings" 
;;   "How are you today? - john doe -" "smtp.asite.com" "jdoe" "secret") 

;; This logs in to the server, tries to authenticate using the username 'jdoe' and password 'secret' (if supplied), 
;; and sends an email with the format: 
;; <pre> 
;;  From:    jdoe@asite.com 
;;  To:      somebody@isp.com 
;;  Subject: Greetings 
;;  Message: How are you today? - John Doe - 
;; </pre> 


(context 'SMTP) 

(set 'debug-flag nil) 

(define (send-mail mail-from mail-to mail-subject mail-body SMTP-server user-name password) 
    (and 
        (set 'from-hostname (nth 1 (parse mail-from "@"))) 
        (set 'socket (net-connect SMTP-server 25)) 
        (confirm-request "2") 
        (unless (and (empty? user-name) (empty? password)) 
           (mail-authorize user-name password)) 
        (net-send-get-result (append "HELO " from-hostname) "2") 
        (net-send-get-result (append "MAIL FROM: <" mail-from ">") "2") 
        (net-send-get-result (append "RCPT TO: <" mail-to ">") "2") 
        (net-send-get-result "DATA" "3") 
        (mail-send-header) 
        (mail-send-body) 
        (confirm-request "2") 
        (net-send-get-result "QUIT" "2") 
        (or (net-close socket) true))) 

(define (confirm-request conf)
    (net-receive socket 'recvbuff 256 "\r\n")
    (if debug-flag (println recvbuff) true)
    ; Empty out pipe. According to SMTP spec, last line has valid code.
    ; added for 1.8 for newLISP 9.2.0
    (while (< 0 (net-peek socket))
        (net-receive socket 'recvbuff 256 "\r\n")
        (if debug-flag (println recvbuff)))
    (starts-with recvbuff conf))
 
(define (net-send-get-result str conf) 
   (set 'send-str (append str "\r\n")) 
   (if debug-flag (println "sent: " send-str)) 
   (net-send socket send-str) 
   (if conf (confirm-request conf) true)) 

(define (mail-authorize user-name password) 
   (net-send-get-result 
       (append "AUTH PLAIN " 
               (base64-enc (append "\000" user-name "\000" password))) "235")) 

(define (mail-send-header) 
    (net-send-get-result (append "TO: " mail-to)) 
    (net-send-get-result (append "FROM: " mail-from)) 
    (net-send-get-result (append "SUBJECT: " mail-subject)) 
    (net-send-get-result (append "X-Mailer: newLISP v." (string (nth -2 (sys-info)))))) 

(define (mail-send-body ) 
    (net-send-get-result "") 
    (dolist (lne (parse mail-body "\r\n")) 
        (if (starts-with lne ".") 
            (net-sent-get-result (append "." lne)) 
            (net-send-get-result lne))) 
    (net-send-get-result ".")) 

(define (get-error-text) 
    recvbuff) 

(context 'MAIN) 

;  test

; (set 'SMTP:debug-flag true) 

; (SMTP:send-mail 
;  "from@example.com" 
;  "to@example.com" 
;  "title" 
;  "body" 
;  "smtp.example.com" 
;  "user.name" 
;  "password")) 

; eof 


Tim Johnson
Posts: 253
Joined: Thu Oct 07, 2004 7:21 pm
Location: Palmer Alaska USA

Post by Tim Johnson »

Thanks folks. I will test that out later today. I'm 4 hours "behind" Lutz
(alaska DST), so results will be late.
Tim

Tim Johnson
Posts: 253
Joined: Thu Oct 07, 2004 7:21 pm
Location: Palmer Alaska USA

Post by Tim Johnson »

Here are results:
Case 1) Sending authentication user-name and password to
server requiring authentication:
debug info

Code: Select all

220 sj1-dm103.mta.everyone.net ESMTP EON-AUTHRELAY2 sent: AUTH PLAIN AHRqb2huc29uAGNoaW1heQ== 503 Need HELO/EHLO first
  
Case 2)Sending message thru localhost (no authentication required) with
empty string for username and password

Code: Select all

220 bart ESMTP Postfix (Ubuntu)
- No message sent
Case 3)Sending message thru localhost (no authentication required) with
empty string for username and password - and - the 'unless sexp
commented out

Code: Select all

220 bart ESMTP Postfix (Ubuntu) sent: HELO johnsons-web.com 250 bart sent: MAIL FROM: 250 2.1.0 Ok sent: RCPT TO: 250 2.1.5 Ok sent: DATA 354 End data with . sent: TO: tim@johnsons-web.com sent: FROM: test@johnsons-web.com sent: SUBJECT: Testing Newlisp sent: X-Mailer: newLISP v.9300 sent: sent: Line one line two sent: . 250 2.0.0 Ok: queued as 162BE80C114 sent: QUIT 221 2.0.0 Bye
- I. E. successful transmission
Observation: In addition to the "Need HELO/EHLO" issue, it appears
to me that a nil return from 'unless is breaking out of the 'and sexp?
Inserting some stubs seems to confirm that....
Thanks - this is fun.
Tim

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

Yes, that authentication code is rubbish - who on earth added that? :)

I think there should be a true as the other result of unless:

Code: Select all

 (unless (and (empty? user-name) (empty? password)) 
    (mail-authorize user-name password)
     true)
because if the two strings are empty this returns nil which stops the and.

As for the HELO, perhaps that should be before the authentication:

Code: Select all

(net-send-get-result (append "HELO " from-hostname) "2")
(unless (and (empty? user-name) (empty? password)) 
           (mail-authorize user-name password)
            true)
That seems to work too.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Thanks for testing and correcting, I also added defaults for user-name and password. This way both new parameters can be left out completely if required.

New version for testing here:

http://www.newlisp.org/smtp.lsp

and highlighted here:

http://www.newlisp.org/syntax.cgi?http: ... g/smtp.lsp

Tim Johnson
Posts: 253
Joined: Thu Oct 07, 2004 7:21 pm
Location: Palmer Alaska USA

Post by Tim Johnson »

Good work Lutz!
Works like a charm......

Now I hate to be such a pest, but I've been playing with ftp.
For my needs, it is more efficient to transmit more than one
file on a connection.

I'll post some code (working or not) later today.
Thanks
Tim

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Good work Lutz! Works like a charm......
Actually the new authentication stuff was done by Cormullion and earlier DMI and recently Kyrill have been contributing fixes or identifying problems to make the code more RFC compliant, so thanks to everybody involved :-)

This new version 2.0 of smtp.lsp goes into the next development release and has been updated in the modules section.

Locked