newlispfanclub.alh.net

• bug fixes
• new 'read' hooks into internal source reader

files and changes notes: http://newlisp.org/downloads/development/

no binary installers for this development release

Lutz

ps: I am out of the country for 2 weeks and will not be able to be on the internet on a daily schedule to read email, give support etc.

Lutz,

You are a legend. Thanks!

Jeff

Looks fascinating... Will this function allow me to convert newLISP code into a sequence of tokens (eg like my attempt at tokenizing, tokenizer.lsp)? Eg for formatting source code...?

No, it can either validate the syntactic integrity of a string or create an unevaluated expression. But it will make tokenizing a string easier.

cormullion wrote:Looks fascinating... Will this function allow me to convert newLISP code into a sequence of tokens (eg like my attempt at tokenizing, tokenizer.lsp)? Eg for formatting source code...?

Call me naive, but does parse without the str-break argument not do this?

No, parse is more like split-by-pattern.

Jeff wrote:No, parse is more like split-by-pattern.

If pattern is given. If pattern is not given, it splits by newlisp lexical rules.
To be scrupulous, parse doesn't treat [text]blablabla[/text] as a lexeme (probably it is not a lexeme from interpreter point of view), but all others are working well.

One of the problems with parse is that it doesn't preserve comments. So for pretty-printing or formatting it can be considered a destructive function... :)

There was an issue with colons I seem to remember:
And parse turns quotes into braces, too.

If you are good with regular expressions, you could use the second syntax of 'parse' or the first syntax of 'find-all' with regular expressions, to tokenize, or a combination of both. Only the parse, without any options behaves as Cormullion describes. But 'find-all' is probably better suited, because there the regular expression describes the token itself and not the space in-between, as 'parse' does.

Lutz

I haven't had any luck with recursive regexing in newLISP. What syntax is supported? Standard PCRE with recursive references does not appear to work (at least, for me).