development release newLISP 9.3.1

Lutz · Post by **Lutz** » Sat Feb 09, 2008 12:32 am

• bug fixes
• new 'read' hooks into internal source reader

files and changes notes: http://newlisp.org/downloads/development/

no binary installers for this development release

Lutz

ps: I am out of the country for 2 weeks and will not be able to be on the internet on a daily schedule to read email, give support etc.

Jeff · Post by **Jeff** » Sat Feb 09, 2008 1:47 am

Lutz,

You are a legend. Thanks!

Jeff

cormullion · Post by **cormullion** » Sat Feb 09, 2008 8:28 am

Looks fascinating... Will this function allow me to convert newLISP code into a sequence of tokens (eg like my attempt at tokenizing, tokenizer.lsp)? Eg for formatting source code...?

Jeff · Post by **Jeff** » Sat Feb 09, 2008 2:04 pm

No, it can either validate the syntactic integrity of a string or create an unevaluated expression. But it will make tokenizing a string easier.

Cyril · Post by **Cyril** » Mon Feb 11, 2008 1:43 am

cormullion wrote:Looks fascinating... Will this function allow me to convert newLISP code into a sequence of tokens (eg like my attempt at tokenizing, tokenizer.lsp)? Eg for formatting source code...?

Call me naive, but does parse without the str-break argument not do this?

Jeff · Post by **Jeff** » Mon Feb 11, 2008 2:17 am

No, parse is more like split-by-pattern.

Cyril · Post by **Cyril** » Mon Feb 11, 2008 4:33 am

Jeff wrote:No, parse is more like split-by-pattern.

If pattern is given. If pattern is not given, it splits by newlisp lexical rules.

Code: Select all

> (parse "(sin (+ 1 1.5))")
("(" "sin" "(" "+" "1" "1.5" ")" ")")

To be scrupulous, parse doesn't treat [text]blablabla[/text] as a lexeme (probably it is not a lexeme from interpreter point of view), but all others are working well.

cormullion · Post by **cormullion** » Mon Feb 11, 2008 4:56 pm

One of the problems with parse is that it doesn't preserve comments. So for pretty-printing or formatting it can be considered a destructive function... :)

There was an issue with colons I seem to remember:

Code: Select all

(parse {(define (fred:jim) (println fred jim))})

And parse turns quotes into braces, too.

Lutz · Post by **Lutz** » Mon Feb 11, 2008 8:18 pm

If you are good with regular expressions, you could use the second syntax of 'parse' or the first syntax of 'find-all' with regular expressions, to tokenize, or a combination of both. Only the parse, without any options behaves as Cormullion describes. But 'find-all' is probably better suited, because there the regular expression describes the token itself and not the space in-between, as 'parse' does.

Lutz

Jeff · Post by **Jeff** » Mon Feb 11, 2008 9:14 pm

I haven't had any luck with recursive regexing in newLISP. What syntax is supported? Standard PCRE with recursive references does not appear to work (at least, for me).