Base64 encoder

pjot · Post by **pjot** » Mon Mar 01, 2004 9:54 pm

Hi all,

After all contributions of newdep I couldn't stay behind... ;-) Below my base64 encoder written in newLISP. Have phun with it.

Cheers
Peter.
--------------------------------------------------------------------------

#!/usr/bin/newlisp
;;
;; Base64 converter using newLISP. Tested on Slackware 9.1 with newLISP 7.5.4.
;;
;; Proxy-servers require a base64 encoded "username:password" to pass through.
;;
;; With this encoder you can hack your way out :-)
;;
;;----------------------------------------------------------------------------

;; Setup base64 encode string
(set 'BASE64 "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")

;; Get input from user
(print "Enter a string to convert: ")
(set 'dat (read-line))

;; Initialize result variable to string
(set 'enc "")

;; Mainloop
(while (> (length dat) 0) (begin

;; Find ASCII values
(if (= (length dat) 1)
(begin
(set 'byte1 (char dat))
(set 'byte2 0)
(set 'byte3 0)))
(if (= (length dat) 2)
(begin
(set 'byte1 (char dat))
(set 'byte2 (char dat 1))
(set 'byte3 0)))
(if (> (length dat) 2)
(begin
(set 'byte1 (char dat))
(set 'byte2 (char dat 1))
(set 'byte3 (char dat 2))))

;; Now create BASE64 values
(set 'base1 (/ byte1 4))
(set 'base2 (+ (* (& byte1 3) 16) (/ (& byte2 240) 16)))
(set 'base3 (+ (* (& byte2 15) 4) (/ (& byte3 192) 64)))
(set 'base4 (& byte3 63))

;; Find BASE64 characters
(if (= (length dat) 1)
(begin
(set 'enc (append enc (nth base1 BASE64)))
(set 'enc (append enc (nth base2 BASE64)))
(set 'enc (append enc "=="))
;; Put 'dat' to empty list
(set 'dat "")))
(if (= (length dat) 2)
(begin
(set 'enc (append enc (nth base1 BASE64)))
(set 'enc (append enc (nth base2 BASE64)))
(set 'enc (append enc (nth base3 BASE64)))
(set 'enc (append enc "="))
;; Put 'dat' to empty list
(set 'dat "")))
(if (> (length dat) 2)
(begin
(set 'enc (append enc (nth base1 BASE64)))
(set 'enc (append enc (nth base2 BASE64)))
(set 'enc (append enc (nth base3 BASE64)))
(set 'enc (append enc (nth base4 BASE64)))
;; Decrease 'dat' with 3 characters
(set 'dat (slice dat 3))))
))

;; Print resulting string
(println enc)

;; Exit
(exit)

newdep · Post by **newdep** » Mon Mar 01, 2004 9:56 pm

Hello Pjot,

Great invention ;-) nice work... ill plug it in as a module...

Norman.

Lutz · Post by **Lutz** » Mon Mar 01, 2004 10:20 pm

Thanks Pjot,

a small improvment:

>>>> instead of <<<<
(set 'enc (append enc (nth base1 BASE64)))
(set 'enc (append enc (nth base2 BASE64)))
(set 'enc (append enc "=="))

>>>> you can do this <<<<
(set 'enc (append (nth base1 BASE64) (nth base2 BASE64) "=="))

in newLISP in all places where it makes sense you can specify more than one arg. Makes the code shorter and faster.

Lutz

pjot · Post by **pjot** » Mon Mar 01, 2004 10:31 pm

Hi Lutz,

Thanks for the tip! I did not know that, I will change it here in my code. (I am sorry for the layout of the program, but the tabs do not seem to appear....)

A decoder will follow.

Regards

Peter.

Lutz · Post by **Lutz** » Mon Mar 01, 2004 10:35 pm

thanks Peter, our pops3.lsp users will be delighted, I will put your code in a future base64.lsp module in the newLISP distribution, if this is Ok with you.

Lutz

pjot · Post by **pjot** » Tue Mar 02, 2004 8:44 pm

Hi Lutz,

Of course, it's ok.

Peter.

pjot · Post by **pjot** » Tue Mar 02, 2004 9:50 pm

As promised, the Base64 decoder below. I've applied your tip to this code but it does not improve readability...

Regards

Peter

-----------------------------------------------------------

#!/usr/bin/newlisp
;;
;; Base64 decoder
;;
;; Due to the nature of newLISP, this is the smallest
;; BASE64 decoder I've ever written.
;;
;;-------------------------------------------------------

;; Setup base64 string
(set 'BASE64 "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/")

;; Get input from user
(print "Enter a string to convert: ")
(set 'dat (read-line))

;; Initialize result variable to string
(set 'res "")

;; Mainloop
(while (> (length dat) 0) (begin

;; Find the indexnumber in the BASE64 definition
(set 'byte1 (find (nth 0 dat) BASE64))
(if (= byte1 nil)(set 'byte1 0))
(set 'byte2 (find (nth 1 dat) BASE64))
(if (= byte2 nil)(set 'byte2 0))
(set 'byte3 (find (nth 2 dat) BASE64))
(if (= byte3 nil)(set 'byte3 0))
(set 'byte4 (find (nth 3 dat) BASE64))
(if (= byte4 nil)(set 'byte4 0))

;; Recalculate to ASCII value
(set 'res (append res (char (+ (* (& byte1 63) 4) (/ (& byte2 48) 16))) (char (+ (* (& byte2 15) 16) (/ (& byte3 60) 4))) (char (+ (* (& byte3 3) 64) byte4))))

;; Decrease string with 4
(set 'dat (slice dat 4))))

;; Print resulting string
(println res)

;; Exit
(exit)

newdep · Post by **newdep** » Tue Mar 02, 2004 10:42 pm

Hello pjot,

Your going too fast :)

Regards, Norman.

Lutz · Post by **Lutz** » Wed Mar 03, 2004 12:18 am

thanks Peter, getting shorter and faster, I wonder if anybody can top this ;-) ?

Lutz

HPW · Post by **HPW** » Wed Mar 03, 2004 7:19 am

Testing on WIN with this sample code:
(PS: BASE64 is a protected symbol!)

Code: Select all

;; 
;; Base64 converter using newLISP. Tested on Slackware 9.1 with newLISP 7.5.4. 
;; 
;; Proxy-servers require a base64 encoded "username:password" to pass through. 
;; 
;; With this encoder you can hack your way out :-) 
;; 
;;---------------------------------------------------------------------------- 

;; Setup base64 encode string 

(context 'BASE64)



(define (encode dat)

(set 'base64charset "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/") 

;; Get input from user 
;(print "Enter a string to convert: ") 
;(set 'dat (read-line)) 

;; Initialize result variable to string 
(set 'enc "") 

;; Mainloop 
(while (> (length dat) 0)
	(begin 
	;; Find ASCII values 
	(if (= (length dat) 1) 
	(begin 
	(set 'byte1 (char dat)) 
	(set 'byte2 0) 
	(set 'byte3 0))) 
	(if (= (length dat) 2) 
	(begin 
	(set 'byte1 (char dat)) 
	(set 'byte2 (char dat 1)) 
	(set 'byte3 0))) 
	(if (> (length dat) 2) 
	(begin 
	(set 'byte1 (char dat)) 
	(set 'byte2 (char dat 1)) 
	(set 'byte3 (char dat 2)))) 
	
	;; Now create BASE64 values 
	(set 'base1 (/ byte1 4)) 
	(set 'base2 (+ (* (& byte1 3) 16) (/ (& byte2 240) 16))) 
	(set 'base3 (+ (* (& byte2 15) 4) (/ (& byte3 192) 64))) 
	(set 'base4 (& byte3 63)) 
	
	;; Find BASE64 characters 
	(if (= (length dat) 1) 
	(begin 
	(set 'enc (append enc(nth base1 base64charset)(nth base2 base64charset)"=="))
	
	;; Put 'dat' to empty list 
	(set 'dat ""))) 
	(if (= (length dat) 2) 
	(begin 
	(set 'enc (append enc (nth base1 base64charset)(nth base2 base64charset)(nth base3 base64charset)"=")) 
	
	;; Put 'dat' to empty list 
	(set 'dat ""))) 
	(if (> (length dat) 2) 
	(begin 
	(set 'enc (append enc (nth base1 base64charset)(nth base2 base64charset)(nth base3 base64charset)(nth base4 base64charset))) 
	;; Decrease 'dat' with 3 characters 
	(set 'dat (slice dat 3)))) 
	)) 

;; Return resulting string 
enc)








;; 
;; Base64 decoder 
;; 
;; Due to the nature of newLISP, this is the smallest 
;; BASE64 decoder I've ever written. 
;; 
;;------------------------------------------------------- 

(define (decode dat)

;; Setup base64 string 
(set 'base64charset "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/") 

;; Get input from user 
;(print "Enter a string to convert: ") 
;(set 'dat (read-line)) 

;; Initialize result variable to string 
(set 'res "") 

;; Mainloop 
(while (> (length dat) 0)
	(begin 
	;; Find the indexnumber in the base64charset definition 
	(set 'byte1 (find (nth 0 dat) base64charset)) 
	(if (= byte1 nil)(set 'byte1 0)) 
	(set 'byte2 (find (nth 1 dat) base64charset)) 
	(if (= byte2 nil)(set 'byte2 0)) 
	(set 'byte3 (find (nth 2 dat) base64charset)) 
	(if (= byte3 nil)(set 'byte3 0)) 
	(set 'byte4 (find (nth 3 dat) base64charset)) 
	(if (= byte4 nil)(set 'byte4 0)) 

	;; Recalculate to ASCII value 
	(set 'res (append res 
		(char (+ (* (& byte1 63) 4) (/ (& byte2 48) 16)))
		(char (+ (* (& byte2 15) 16) (/ (& byte3 60) 4)))
		(char (+ (* (& byte3 3) 64) byte4))))

	;; Decrease string with 4 
	(set 'dat (slice dat 4)))) 

;; Print resulting string 
(trim res "\000")
)

(context 'MAIN)

Bugs corrected!

newdep · Post by **newdep** » Wed Mar 03, 2004 3:32 pm

Hello HPW,

"\000" odd actualy... Looks like windows displays the <EOF> or <EOL>
which is an NULL-byte normaly but im not sure on this...

Norman.

Lutz · Post by **Lutz** » Wed Mar 03, 2004 4:08 pm

newLISP will only display the null character if it was somehow specified, internally there is one more character behind it, i.e:

(set 'var "\000\000\000") => "\000\000\000"

(length var) => 3

but internally newLISP allocates one more byte, which is always \000 this way strings are limited when using 'print', 'format' etc. i.e.

(set 'var "A\000\000") => "A\000\000"

but

(println var)
A
"A\000\000"

the first A is the printed one and the other thing is the return value from the print statment. Internally the var string is allocated with 4 bytes.

most of newLISP's functions can work on binary content allthough not documented spefically because strings are always stored in a buffer with a zero appended and the buffer length stored in a different field of the LISP cell.

Lutz

Lutz · Post by **Lutz** » Wed Mar 03, 2004 6:10 pm

Perhaps Peters BASE64 code is just fine, from:

http://www.securecode.net/Base64Convert+main.html

>>>At the end of the encoding process we might run into a problem. If the size of the original data in bytes is a multiple of three, everything works fine. If it is not, we might end up with one or two 8-bit bytes. For proper encoding, we need three bytes, however.

The solution is to append enough bytes with a value of '0' to create a 3-byte group. Two such values are appended if we have one extra byte of data, one is appended for two extra bytes.
>>>

Si I think you can just strip of the trailing \000 doing a: (trim str)

Lutz

pjot · Post by **pjot** » Wed Mar 03, 2004 7:58 pm

Hi all,

Indeed it is the nature of Base64 which is bothering us, I think. For the ENCODING part, we are going from 3 bytes to 4 'bytes' (3x8 to 4x6). The decoding part is the reverse (4x6 to 3x8).

The padding sign '=' is used to fill up the empty places when the input string originally is not a multiple of 4. This '=' sign however is not part of the BASE64 string, therefore with the decoder a '0' is produced in order to perform a succesfull binary calculation backwards. This might result in a string with "\000" at the end.

Even not shown in my prompt here, the length of the encoded "Peter" = "UGV0ZXI=" will deliver a decoded length of 6 which is "Peter\000" again, since the base64 string ends with the padding symbol '='.

Indeed the best workaround for all this is to TRIM your resulting string in the BASE64 decoder function towards a regular one. (How is this with other languages? Gnu AWK and Scriptbasic happen to do this automatically.)

So the last line in the decoder must be:

;; Print resulting string
(trim res "\000"))

Thanks for the tip.

Peter.

Lutz · Post by **Lutz** » Wed Mar 03, 2004 8:05 pm

GNU Awk and Script Basic only handle 'C' strings which always end with a zero. These languages cut of anything after a zero implicitly, as a drawback they don't handle binary data as you can using newLISP.

Lutz

HPW · Post by **HPW** » Wed Mar 03, 2004 8:09 pm

It seems that my test with my 10 KB string failed, because
it does not work with '\r\n'

This works:

> (setq a(BASE64:encode "Test\nTest"))
"VGVzdApUZXN0"
> (BASE64:decode a)
"Test\nTest"
> (setq a(BASE64:encode "Test\rTest"))
"VGVzdA1UZXN0"
> (BASE64:decode a)
"Test\rTest"

But:

> (setq a(BASE64:encode "Test\r\nTest"))
"dA=="

pjot · Post by **pjot** » Wed Mar 03, 2004 8:15 pm

Hi Lutz,

Well, I mentioned the other languages because I did not experience this problem there... I am sorry if it appeared as a criticism, that really was not my intention... indeed newLISP is functioning more consequent since it will return everything the program has asked for.

HPW: I look into that problem now, just a minute...

P.

pjot · Post by **pjot** » Wed Mar 03, 2004 8:20 pm

Hi HPW,

In my Linux environment this is convertable without any problem. My self-defined function delivers this result with "Test\r\nTest":

VGVzdA0KVGVzdA==

I will literally cut and paste your program now.

Lutz · Post by **Lutz** » Wed Mar 03, 2004 8:20 pm

don't worry, i didn't take it as criticism, i just take all opportunity to explain things about newLISP knowing that many people read this board

Lutz

pjot · Post by **pjot** » Wed Mar 03, 2004 8:34 pm

Hi HPW,

I just found the problem. There is an error in your corrected encoding part. Look at this part in your BASE64:encode:

;; Find BASE64 characters
(if (= (length dat) 1)
(begin
(set 'enc (append(nth base1 base64charset)(nth base2 base64charset)"=="))

This should be:

;; Find BASE64 characters
(if (= (length dat) 1)
(begin
(set 'enc (append enc (nth base1 base64charset)(nth base2 base64charset)"=="))

Then it works.
Btw: nice to see how contexts are working, I did not experiment with them yet!

Regards

Peter.

HPW · Post by **HPW** » Wed Mar 03, 2004 8:34 pm

Oops! Typo.

When changing to Lutz advice I change to

(set 'enc (append (nth base1 base64charset)(nth base2 base64charset)"=="))

instead of

(set 'enc (append enc (nth base1 base64charset)(nth base2 base64charset)"=="))

Sorry for that.

HPW · Post by **HPW** » Wed Mar 03, 2004 8:35 pm

Thanks, you was faster! :-)

pjot · Post by **pjot** » Wed Mar 03, 2004 8:41 pm

;-) ok ok... no problem! Thanks for your input!

HPW · Post by **HPW** » Wed Mar 03, 2004 8:46 pm

A little benchmark test with 50KB string:

> (time(setq a (BASE64:encode bigtxt)))
2391
> (time(setq b(get-string(hpwMimeEncodeString bigtxt))))
0
> (length a)
66696
> (length b)
68450
> (length bigtxt)
50022
>

pjot · Post by **pjot** » Wed Mar 03, 2004 8:50 pm

hmmm... so your mime encode is faster, but also appears to produce a LONGER result? Shouldn't the result of the conversion be of the same length?