Compression

newdep · Post by **newdep** » Thu Nov 24, 2005 12:11 pm

Hi Lutz,

Its on my wishlist for some time now but how do you think about a small and
simple compression/decompression implementation in newlisp? (like a small lha implementation) someehing like: (compress str) and (decompress str).

Regards, Norman.

HPW · Post by **HPW** » Thu Nov 24, 2005 12:33 pm

There was a thread about importing gzip:

http://www.alh.net/newlisp/phpbb/viewto ... light=zlib

And for windows (hpwNLUtility):

http://www.alh.net/newlisp/phpbb/viewto ... light=zlib

newdep · Post by **newdep** » Thu Nov 24, 2005 1:12 pm

Your right about the thread ;-) Though I never came to the Zlib because
it simple too big for a simple compression. I also had a look at miniLha lib
but that one still is 16 Kb..
(I dont care what compression as long as it compresses :-)

How about a simple LZW imlant?

Though Im hoping of a C implemented fast simple compression routine
inside newlisp. the CRC was also done by Lutz and its small and fast.

Perhpas Lutz can do some X-mass features ;-)

Regards, Norman

pjot · Post by **pjot** » Thu Nov 24, 2005 7:59 pm

Do you want compression for text files? Or also binary files?

If only textfiles must be compressed, you could write a compression function yourself in newLisp. This should be fast enough, even for large ASCII files.

Peter

newdep · Post by **newdep** » Thu Nov 24, 2005 8:38 pm

compression is compression so that includes it all ;-)
No bitstuffing or mapping on ascii layers, but a real compression algorithm..

But I wont spoil the fun, if you have a newlisp version for ascii drop it in ;-)

Regards, Norman.

newdep · Post by **newdep** » Mon Nov 28, 2005 6:18 pm

So its finaly Done, we have compression, done by Pjot!

You can download it from his site at -> http://www.turtle.dds.nl/newlisp/lzw.lsp

PS: its not the worldrecord on speed, thought it is compession! ;-)

pjot · Post by **pjot** » Mon Nov 28, 2005 11:13 pm

Speed is a problem, that's true. But there may be some improvements on the code which also may improve speed.

During this week I'll finalize the program. And let you know (of course).

Peter

(PS you can also buy a faster computer.... ;-) )

pjot · Post by **pjot** » Wed Nov 30, 2005 9:17 pm

The LZW compression context has been optimized and is ready for consumption. Compared to the first implementation, the compression speed has improved with 25%, while the decompression speed has improved dramatically with about 80%.

LZW is a technique which normally is used to compress GIF and TIFF files. It is very suitable to compress text. Files up to 500k compress reasonably fast with this context.

http://www.turtle.dds.nl/newlisp/lzw.lsp

If somebody has suggestions to increase the compression speed even more, please let me know. :-)

Enjoy,

Peter

pjot · Post by **pjot** » Mon Mar 13, 2006 12:02 pm

The LZW compression has been optimized even more. The average compression speed compared to the former situation is 5x faster.

On my P4 2Ghz an e-text of 500kB can be compressed in 4 seconds now. If your machine is faster then it will take even less time.

The latest version is here:

http://www.turtle.dds.nl/newlisp/lzw.lsp

To test, just copy some file into the same directory as 'LZW.lsp', with the name "dump.txt", and run 'LZW.lsp'. The compression also works with binary files.

Regards
Peter

newdep · Post by **newdep** » Mon Mar 13, 2006 12:54 pm

On an AMD 64 3200+

filesize: 4614809 (pure text, dictionary file)
Starting compression, please wait... Time spent (msecs): 17020
Starting decompression, please wait... Time spent (msecs): 30283

1/4 of 4614809 (1153703 bytes or the same file) ->
Starting compression, please wait... Time spent (msecs): 4282
Starting decompression, please wait... Time spent (msecs): 7589

They both come pretty close to a 4:1 ...

Norman.

pjot · Post by **pjot** » Fri Mar 17, 2006 10:13 pm

Self-extracting possibilities were added.

http://www.turtle.dds.nl/newlisp/lzw.lsp

Just call

(LZW:lzx "myfile.ext")

and it will create a self-extracting ".lzx" file. If this ".lzx" file is executed by newlisp, it will be decompressed. In Unix/Linux it is also possible to run it directly, like "./myfile.ext.lzx". In Windows you need to execute "newlisp myfile.ext.lzx" to decompress.

Peter

tom · Post by **tom** » Fri Mar 17, 2006 11:21 pm

this needs to go in a backup/restore script. I've been using back2cd.tcsh, a script that a guy in my lug wrote, but I think that newlisp could do at least as well, especially with lzw.lsp around...

tom · Post by **tom** » Sun Mar 19, 2006 11:16 am

hey peter, stupid question:

how do you give it the file to compress?

$./lzw.lsp

gives your error,

ERROR: File not found! Exiting...

I did say up front this was a stupid question :-)

thanks.

pjot · Post by **pjot** » Sun Mar 19, 2006 12:24 pm

Hi Tom,

In the MAIN context of "LZW.lsp" the demonstration is looking for a file called "dump.txt". Just create a file with this name in the same directory and run. Currently the LZW creates a self extracting . lzx file, for which you need newLisp to decompress.

Peter