Different Behavior, Script vs. Compiled/Linked

[semperos]

First, a big thanks to Lutz and the community for creating newLISP. I'm new to newLISP.

I've written a simple grep-like script using newLISP. The code is here:

https://gist.github.com/2041991

Used like

Code: Select all

mgrep "foo||bar" *.txt

As a script, this code works as expected; it reads files in line-by-line, printing only those lines that match both "foo" and "bar". However, when I compile and link this (so it's all self-contained), I get strange behavior.

Various things happen. Sometimes it works as expected; sometimes it drops down to a newLISP REPL instead of printing results; sometimes it complains that Lisp code that I'm read-lining through has undefined functions in it (!).

Can anyone shed light on any one of these issues? Why such a difference between a "script" version and a compiled version?

[semperos]

Forgot to include my computer specs:

• newlisp --version => "newLISP v.10.4.0 on OSX IPv4/6 UTF-8"
• Mac OSX Lion

[cormullion]

I get the same result here. (I think. I was convinced for a while that one attempt worked.) There was an issue with 32 and 64 bit, but I think Lutz fixed that...

[Lutz]

I cannot repeat that error, no matter if using the 32-bit or 64-bit of newLISP. Can you give exact instructions, how the the link was made and the resulting linked script is used? I am also on OSX Lion.

Also:

There should be no space in here "#! /usr/bin/newlisp", instead "#!/usr/bin/newlisp".
Then on UNIX systems, like Mac OSX, you can just execute mgrep by itself after making it executable with "chmod 755 mgrep", no linking required.

link.lsp doesn't compile anything, it just takes the source script, encrypts it and appends it to the end of a newLISP executable. When linking, make sure you have a newLISP executable and the source in the current directory. When instead using pathnames to point to different locations, make sure you don't point to linked versions of files. Like with a raw script, you have to make it executable.

[semperos]

Thanks for addressing this, Lutz.

I cannot repeat that error, no matter if using the 32-bit or 64-bit of newLISP. Can you give exact instructions, how the the link was made and the resulting linked script is used? I am also on OSX Lion.

Here are the directories I'm working with:

Code: Select all

~/dev/newlisp/utils/
~/dev/newlisp/utils/build/

The `utils` folder contains my source file, `mgrep.lisp`. The `build` folder contains my newLISP executable and the `link.lsp` file. When I'm ready to build a self-contained executable from my source file, I copy it into the `build` folder so that the newLISP executable, the `link.lsp` file, and my source file are in the same directory.

I wrote one extra function into my `link.lsp` file (code here) so I can just write:

Code: Select all

(program "mgrep.lisp")

...and it evaluates as:

Code: Select all

(link "newlisp" "mgrep" "mgrep.lisp")

So when I run `newlisp link.lsp` at the command-line and then run my `program` function, the output is:

Code: Select all

original newlisp executable:newlisp
new executable:mgrep
source:mgrep.lisp
true

There should be no space in here "#! /usr/bin/newlisp", instead "#!/usr/bin/newlisp".
Then on UNIX systems, like Mac OSX, you can just execute mgrep by itself after making it executable with "chmod 755 mgrep", no linking required.

Noted, I'll fix the space. It runs fine as a script regardless, which is why I have this question. I want to make executables so I can share my utilities with other folks without them having to install newLISP.

link.lsp doesn't compile anything, it just takes the source script, encrypts it and appends it to the end of a newLISP executable. When linking, make sure you have a newLISP executable and the source in the current directory. When instead using pathnames to point to different locations, make sure you don't point to linked versions of files. Like with a raw script, you have to make it executable.

Understood. I'm pretty sure I hit all your points in my above process, and as I mentioned, it works under certain circumstances. Thanks again for the help.

[cormullion]

On my machine I think it might be something to do with my PATH. Not sure. But it works better when I call the binary with its full name:

Code: Select all

$ mgrp "setq" *.lsp
$                                      <--- no results
$ ~/bin/mgrp "setq" *.lsp
a-newlispdoc.lsp/344: 		(if source-link (setq link (string {<a href="} filename 
recursive-descent-parser.lsp/6: ;(setq *token* '( 3 ^ 5 ^ 7 + 5 * 3 + 7 / 11))
recursive-descent-parser.lsp/99: (setq *token* '(3 ^ 5 ^ 7 + 5 * 3 - 7 / 11))
...
$ 

[Lutz]

Just execute the command 'set' in the terminal shell without any arguments, then look for the definition of 'PATH'.

If the path to '/Users/cormullion/bin' is missing you could create a .profile file with this:

Code: Select all

export PATH=$PATH:/Users/cormullion/bin

.profile gets read when you open a shell on OSX.

[cormullion]

Hmm, odd. I think my path includes the bin directory. That's why it can find 'mgrp' when I'm not in '~/bin:

Code: Select all

$ cd
iMac:~ cormullion$ pwd
/Users/cormullion
iMac:~ cormullion$ mgrp
newLISP v.10.4.0 on OSX IPv4/6 UTF-8, execute 'newlisp -h' for more info.

> (exit)
iMac:~ cormullion$ mgrp "set" *.lsp
newLISP v.10.4.0 on OSX IPv4/6 UTF-8, execute 'newlisp -h' for more info.

> (exit)
iMac:~ cormullion$ 
$ cd ~/bin
iMac:bin cormullion$ mgrp "set" *.lsp
mgrep.lsp/12:     (set 'prog-args (rest (rest (main-args))))
mgrep.lsp/13:     (set 'prog-args (rest (main-args))))
mgrep.lsp/16:   (set 'line-count 0)
iMac:bin cormullion$

Odd how it runs correctly only when I'm in ~/bin.... Perhaps it's more of Apple's weird sandboxing...

[Lutz]

Are you using perhaps a special shell on OSX?

Also can you do a 'which mgrep' ?

Perhaps a different mgrep is installed?

[semperos]

My problem still persists. Lutz - Do you do anything different than what's described in the manual for compilation? Do you have any info on the other behaviors I mentioned above? (that running the compiled version over Lisp code files gave complaints about undefined functions in the code I was read-lining in; and the issue of my compiled version dropping down to a REPL sometimes when it fails?)

In case your last questions were also directed to me, I am successfully able to put `mgrep` on my path, in which case a `which mgrep` points to my compiled newLISP script.

[Lutz]

I created the same directory structure as in your post and repeated your course of actions.

~/dev/newlisp/utils/
~/dev/newlisp/utils/build/

… trying to recreate the same conditions, including using your link.lsp with your added 'program' function, but could not break it trying many different mgrep's on different text files. I am running Mac OSX Lion 10.7.3 and have been using the original newLISP 10.4.0 as found on the download page.

I also retested on UBUNTU 11.10 and on Windows XP. On both OS the 'program' gives problems but using the normal (link "new lisp" "mgrep" "mgrep.lsp") and (link "newlisp.exe" "mgrep.exe" "mgrep.lsp") produce correctly working mgrep(.exe) programs.

[semperos]

Then I officially ask you to stop helping me on this front :-)

I'm really impressed by the efforts you've taken to reproduce my problem, and I sincerely appreciate the help. I will continue to tinker with the compilation.

On a slightly unrelated note - is there a syntax for the default globbing support for scripts (which I'm using in mgrep) that will recursively descend into an arbitrary number of directories? Something like the Ruby **/** type syntax?

[semperos]

Ok, perhaps I lied, this is just too curious. Lutz - are you moving the compiled file from where you build it? Are you putting it somewhere else on your path and using it elsewhere in your file system?

Upon further inspection, I'm seeing the same behavior cormullion is. I stopped using my `program` fn, just doing a standard compile. I then make it executable and put it in my ~/bin, which is on my path (and `which mgrep` returns that file path).

If I go into my home directory and do `mgrep "foo" *` I get the following:

Code: Select all

ERR: invalid function : (TeX Live 2011)

...because I happen to have a file that has that text in it, parentheses and all.

On the other hand, if do ~/bin/mgrep, it works as expected, returning results instead of that error.

Weird, no? I can confirm this happens for me on Mac OSX Lion and Ubuntu 11.10. Thoughts?

[Lutz]

Yes, I see now the same effect on OSX and other UNIX, if the generated mgrep is not in the current directory. On Windows XP, I can move mgrep into a different directory in the executable path and it will still work. On LINUX and FreeBSD is also works if prefixing mgrep with the full path. But that does not work on OSX.

There is no quick fix for this in UNIX. Windows has an API to get the directory of the executable running, which newLISP uses. On UNIX there seems to be no such function. Searching through the environment path is too involved.

link.lsp is popular on Windows. On UNIX people are accustomed to use the UNIX script way with "#!/usr/bin/newlisp" or "#!/usr/bin/env newlisp" in the script. Because of this, this problem has not surfaced earlier.

Therefor link.lsp should only be used on Windows or perhaps I will eliminate it completely. On Windows it is possible to write .cmd files which work similar to UNIX script files.

[cormullion]

I'm glad it wasn't me going completely mad... :) I rarely use this link option, so it's more of an academic issue.

[semperos]

I'm glad I'm not going mad, either :-)

The only reason I tried it was that I was trying out a few new Lisps (Clojure is my primary language) and was interested in platform executable compilation, as Clojure only compiles to Java artifacts that require a bit more setup to be used as scripts.

Thanks for all the help!

[cormullion]

I suspect there's little speed difference for shorter scripts between linked executables and plain scripts. Also, it's much easier to update and modify scripts when they're independent of the newLISP binary. (There's also the slight risk of losing the original source... :)

I can see that the feature of bundling everything together is useful, though.

[Lutz]

Again: link.lsp doesn't compile anything, it just takes the source script, encrypts it and appends it to the end of a newLISP executable. So speed wise there is no difference in execution of the script, linked or not.

[semperos]

Yep, understood on how link.lsp works. newLISP is plenty fast; my only goal was to create self-contained and portable utilities to share between identical machines (at my work).

[Lutz]

Not sure how your work machines are configured, but if they are OSX/UNIX machines, all with user writable $HOME/bin directories, you could write a small shell script copying the newlisp executable and script over the office network. This way you only need to distribute one file initially.