load lisp-files from inside wrapped EXE

HPW · Post by **HPW** » Wed Oct 15, 2003 5:35 am

Hello Lutz,

Is it possible to access lisp-files packed into the virtual file system
inside the packed EXE?

I know your tip with link.lsp to attach a Lisp-file, but I want to pack a large
number of files (when possible) with a directory-structure inside it and read from there?

Lutz · Post by **Lutz** » Wed Oct 15, 2003 10:03 pm

Yes, you would use 'load', image the following directory structure

c:\bwidget\*.tcl
c:\myextension\*.tcl
c:\freewrap
c:\freewrap\newlisp-tk.tcl
c:\freewrap\mystuff.lsp
c:\freewrap\images\*.gif
c:\freewrap\bwidget.txt
c:\freewrap\myextension.txt
c:\freewrap\mylisptstuff.txt

content of file 'mylispstuff.txt':
>>>>
/freewrap/mystuff.lsp
>>>>

then wrap:

freewrap newlisp-tk.tcl -f images.txt -f bwidget.txt -f myextension.txt -f mylispstuff.txt

in the config file 'newlisp-tk.config' put the following option:

set Ide(initCommand) "wm withdraw . ; NewlispEvaluate {(load \"/freewrap/mystuff.lsp\")}"

the first statement before the semicolon 'wm withdraw .' will supress the tk console windows coming up, so you will only see window stuff you are bringing up yourself. The command after the semicolon: NewlispEvaluate {(silent (load \"/freewrap/mystuff.lsp\"))} is a tk procedure telling newLISP to load your file.

See also the manual section: Delivering Apllications with newLISP-tk

This mixed newLISP/TclTk stuff is a bit tricky with all those 100,000 different string delimeters {,", which also have different meaning in newLISP and TclTk :=), but that's what you gotta do ;)

Instead of doing the "set set Ide(initCommand) " stuff you also could put at the very end of the newlisp-tk.tcl file:

NewlispEvaluate {(silent (load "/freewrap/mystuff.lsp"))}

That would be a little bit less cryptic, but still leave this in you newlisp-tk.config file:

set Ide(initCommand) "wm withdraw ."

(don't forget the dot)

let us know when it works

Lutz

HPW · Post by **HPW** » Thu Oct 16, 2003 6:16 am

Hello Lutz,

Thanks for the informations!

I think it does not what I want. So look at my try:

Content of /freewrap/lisp/mylisp.lsp: (Also wrapped inside a new TK-EXE)

(setq hpwtest "9999")

newLISP v7.2.2 Copyright (c) 2003 Lutz Mueller. All rights reserved.
(load "/freewrap/lisp/mylisp.lsp")
true
> hpwtest
"9999"
> (setq hpwtest nil)
nil
> hpwtest
nil
>(tk "NewlispEvaluate {(load \"/freewrap/lisp/mylisp.lsp\")}")
""
> true
> hpwtest
"9999"
> (setq hpwtest nil)
nil
>

Now renamimng mylisp.lsp to something other

> (tk "NewlispEvaluate {(load \"/freewrap/lisp/mylisp.lsp\")}")
""
> nil
> hpwtest
nil

So when I see it correct, the call on the TCL-function does it right, but did not look inside the wrapped EXE. So how to tell the LOAD-command where to search? Or must there be a new TCL-command for extracting the wanted file to temp and newlisp load it from there?

HPW · Post by **HPW** » Thu Oct 16, 2003 6:36 am

Also when I try the first hint:

set Ide(initCommand) "wm withdraw . ; NewlispEvaluate {(load \"/freewrap/lisp/mylisp.lsp\")}"

I get this:

Error sourcing /Freewrap/newlisp-tk.tcl: can not find channel named "0"

Lutz · Post by **Lutz** » Thu Oct 16, 2003 1:29 pm

I should have tried it first myself ;-), sorry about it, but I found the solution now and tried it out.

What went wrong:

It didn't work because newLISP, which lives outside the wrapped application, of course doesn't know about the freewrap virtual filesystem.

Also the set 'Ide(initCommand)' may not contain any newLISP code because when 'Ide(initCommand)' is executed from inside newlisp-tk.tcl, the communications channels to newLISP are not established yet, this is why you get a : can not find channel named "0".

This works:

I tried it out! after having wrapped tcltk-app.lsp, I tried the following in the newLISP-tk console window.

(tk "set fle [open /freewrap/tcltk-app.lsp r]; NewlispEvaluateBuffer [read $fle] 0")

TclTk opens the file than reads and evaluates it with NewlispEvaluateBuffer', which is a function internally used by newlisp-tk.tcl to send the edit-window to newLISP for evaluation when you click the evaluation button in the edit browser.

The last parameter set to 0 in the example tells newLISP to do a 'silent' evaluation, without seeing the evaluation result in the console.

After the first experiment went well, I pasted the following line as the last line into newlisp-tk.tcl:

set fle [open /freewrap/tcltk-app.lsp r]; NewlispEvaluateBuffer [read $fle] 0

Then wrapped again. As expected the tcltk-app.lsp (example from the windows distribution) went up without a problem. So this seems the way to go.

Lutz

HPW · Post by **HPW** » Thu Oct 16, 2003 1:58 pm

Thanks Lutz,

now it works as expected and real smart with the TK-Buffer.
Powerfull function with a lot possibilitys!

HPW · Post by **HPW** » Thu Oct 16, 2003 2:19 pm

Start thinking with the new possibilitys:

For example file checking:

(tk "file exists /freewrap/lisp/mylisp.lsp")

returns "1" when file is inside wrapped EXE "0" if not.

2 powerfull programming languages in one application combined
with a simple, but effektiv interface is really smart.

HPW · Post by **HPW** » Thu Oct 16, 2003 2:58 pm

There comes a question up:

Is the complete wrapped EXE loaded into memory?
When I wrapp a lot of files inside the EXE will the startup slow down?
I think this is worth some tests.

Starting an empty newlisp set up 2 process with 5.75 MB and 1.85 MB.
So lets wrap!

HPW · Post by **HPW** » Sat Oct 18, 2003 3:48 pm

I did a heavy wrapping test with a complete database with 16600 Lisp-Files from a 5.7 MB ZIP.
Unzipped 11.8 MB on Disk (69.4 MB on NTFS because cluster size)

Now my newlisp-tk.exe is 9.85 MB.

>Is the complete wrapped EXE loaded into memory?

Yes it seems so, memory load is about the difference higher.

>When I wrapp a lot of files inside the EXE will the startup slow down?

Only a little bit in the range about 1-2 sec.

Starting the empty newlisp set up 2 process with 9.3 MB and 2.0 MB.
Compression-ratio seems a little better then standard WINZIP.

Now remains the question if lisp-loading from the preloaded EXE (RAM) is faster than from disk-files.

Now it can be optimised to put absolutly needed file in the EXE and the other content dynamicly loaded from an ZVFS-Zip.

Lutz · Post by **Lutz** » Mon Oct 20, 2003 12:22 am

sounds very impressive, I wonder what kind of application you are developing?

Lutz

HPW · Post by **HPW** » Mon Oct 20, 2003 5:46 am

> .. I wonder what kind of application you are developing?

It's not a new application, it will be a port of an autolisp-app running under autocad. Ths lisp-files comes from a oracle-database with a complete ERP-dataset for calculating sales orders. It is used in an autocad-oem app and is running for some years now. It is read only and running interpreted with dynamic caching. So even under interpreted autolisp it's performance is OK. It use dynamicly lisp-generated interfaces like TK. So now I want a self-contained standalone-app without the pricy oem-license with the identical logic (without 3D-graphics of cource). Multiplatform is also wellcomed (Therefor lisp-tables, no platform dependent database). A second step would be a server-based solution with the same logic.

It seems so far that it could be done, when I solve my problem with formula's from the ERP-system. Therefor I asked for the infix parser, to be able to process them under newlisp. Everything else should be possible under newlisp so far.

Lutz · Post by **Lutz** » Mon Oct 20, 2003 4:40 pm

Its good to hear how newLISP can solve real world problems. Many people think LISP is just for academics.

I just finished an infix/postfix/prefix parser in newLISP and have published it on http://newlisp.org/news . The algorithm is sound and I have used variations on it it many times, coded in 'C' or in Java. This is the first time I have done it in newLISP, so give it a little bit more testing. Read the news post and the header of the file for more info.

The file is also in the development directory and will be included in the next newLISP release.

Lutz

HPW · Post by **HPW** » Mon Oct 20, 2003 5:16 pm

Hello Lutz,

Thank you very much for your fast and great work.
And to see how incredible small it is, is still unbelievebel.

>Its good to hear how newLISP can solve real world problems.

After the great developments in last month, I now believe that it able to do solve such problems. It only has to show the final overall performance and the robustness and simplicity of TK-user-interfaces. This is the remaining matter to test.

Now to the infix-parser:

What I test so far does it right. The only thing remains for me, to replace all operators with surounding spaces. Also I had to replace all "," decimal delimiter to decimal-points, because possible formula-string are like this "(foo+10,3)/2)". But this should be no problem.

Again, thanks a lot.

Back to testing. :-)

HPW · Post by **HPW** » Mon Nov 17, 2003 4:16 pm

With the new 7.3.4 I have no problem to load the huge Lisp-file from a file directly.
It is really fast. But when I try it from the wrapped EXE or file-system through the TK-frontend it seems to hang.
When I use a far more smaller file, it comes back at once.

Code: Select all

(tk "set fle [open /DBA/artbez.lsp r]; NewlispEvaluateBuffer [read $fle] 0")

When I only use the:

Code: Select all

(tk "set fle [open /DBA/artbez.lsp r]")

I get back a filehandle at once.

Can there be a problem with a huge file-buffers and NewlispEvaluateBuffer?

HPW · Post by **HPW** » Mon Nov 17, 2003 4:43 pm

So far a workaround:

Code: Select all

(set 'tmpfile(tk "::freewrap::unpack  /DBA/artbez.lsp C:/temp/"))
(load tmpfile)
(delete-file tmpfile)

This is fast enough.

Lutz · Post by **Lutz** » Mon Nov 17, 2003 6:13 pm

there seems to be a limit of sending bigger than 64k chunks from Tcl/Tk with a 'puts' statement, which normally is enough as it only handles edit buffers from the browser-editor, which typically are only a few kbyte.

I think I can fix that, partitioning the sending of buffers in Tcl, but it will take a while. I am glad you found a workaround meanwhile.

Handling bigger buffers of course would be nice because it would be possible to edit LISP databases in the browser-editor. I will give it a try.

Lutz

HPW · Post by **HPW** » Mon Nov 17, 2003 8:19 pm

>I will give it a try.

I agree with you, it is worth the try.
Would be nice to be able to get the direct way.
So keep on!

Lutz · Post by **Lutz** » Mon Nov 17, 2003 8:41 pm

the limit for TCL/puts on Wndows seems to be ~ 20k, on Linux its ~ 100k

Lutz