Few questions ...

Pondering the philosophy behind the language
Locked
Kazimir Majorinc
Posts: 388
Joined: Thu May 08, 2008 1:24 am
Location: Croatia
Contact:

Few questions ...

Post by Kazimir Majorinc »

I'm trying to write an article on Newlisp macros (mostly stuff I already discussed) and I need few bits of information:

Lutz: did you reinvented macros in the form of fexprs through your own experiments, or you analysed the past and concluded that you liked MacLisp and Interlisp "special forms" more than CL or Scheme. When it happened (year?)

Everyone: theoretically, dynamic scope causes funarg problems, discussed on this forum (itistoday, Jeff recently) and on my blog. (Briefly, if your program uses variable x, and it calls the function or macro that uses local variable x, both variables of the same name are accessible, and if one is not careful, called function can confuse the two. One of the main purpose of the contexts is to solve the problem if it occured.)

Did you have such problem in practice? How serious it was? Did it happened that contexts were not enough for your problem?

Please, answer on this questions, because if you do not comment, I can assume that you had no problems, but I cannot prove it.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Fexprs were in newLISP from the beginning (v.1.0 was released in 1993). I thought that fexprs could do almost anything expansion/rewrite macros could do: creating special forms with non-standard evaluation rules. This is the reason why the name 'define-macro' has 'macro' in it. Its really a function, but its purpose is to let you do, what you normally do with macros.


But rewrite macros are coming to newLISP too:

Using a new 'reader-event' function in newLISP v.10.1.6, rewrite macros can be created. A rewrite function in an ew module macros.lsp packaged with v.10.1.6 intercepts code translation allowing for code tranformations during load of a program.

For details see here: http://www.newlisp.org/macros/

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

newLISP is cool because you can program in a Lisp-like way if you want to, but otherwise it's a powerful and easy to use scripting language without them. I mostly use it in the latter way, and very rarely use macros unless they're essential - ie no other way to do it. I don't think there's a single macro in Dragonfly, and there was only one in my previous blogging app. I think I should be using them in nldb.lsp but can't work out how to do it, so I control evaluations another way...

I'm assuming that the use of contexts (my understanding is outlined here) is sufficient to prevent the problems of confusing identically-named variables (this is what I thought I did here but I might be wrong).

Looking forward to see what the new macros can do...!

Kazimir Majorinc
Posts: 388
Joined: Thu May 08, 2008 1:24 am
Location: Croatia
Contact:

Post by Kazimir Majorinc »

Aha, I see. If I understood well, these macros are similar to C macros in a sense that you explicitly define source and target expression, while other Lisp macros specify the code that expands the expression.

Maybe you could consider some other name, something like expand- (or reader- or rewrite- or preprocessing- or inline-) -patterns because of possible, almost unavoidable confusions with (1) current, very expressive define- and lambda-macros in Newlisp (2) macros in other Lisps specifying the code that calculates target expressons.

Especially if there is a possibility that people might conclude that "Newlisp macros are not as good as X-Lisp macros" while - intentionally or not - stay silent about fexpr-macros. These issues are complex and even very experienced programmers can be confused.

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

While I have mixed feelings about doing things just so that Common Lisp users will be 'happier' and less confused than usual about newLISP, I can also see an argument to naming things precisely...

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

The new 'macro' facfility in upcoming 10.1.6 is actually very similar to macros CL. No mixed feeling regarding naming are necessary :-)

From "The Common LISP Cookbook":
http://cl-cookbook.sourceforge.net/macros.html
A macro is an ordinary piece of Lisp code that operates on another piece of putative Lisp code, translating it into (a version closer to) executable Lisp.
or from: "An Introduction to Common Lisp Macros"

http://www.apl.jhu.edu/~hall/Lisp-Notes/Macros.html
1. Basic Idea: Macros take unevaluated Lisp code and return a Lisp form. This form should be code that calculates the proper value. Example:

(defmacro Square (X)
'(* ,X ,X))

This means that wherever the pre-processor sees (Square XXX) to replaces it with (* XXX XXX). The resultant code is what the compiler sees.
That is similar to what the 'macro' in macro.lsp does:

Code: Select all

(macro (square X) (mul X X)) 

=> (lambda-macro (X) (expand '(mul X X)))

square => (lambda-macro (X) (expand '(mul X X)))

((lambda-macro (X) (expand '(mul X X))) 3) => (mul 3 3) ; the resultant code
'square' now contains the function generating the code the newLISP translater uses to rewrite the code. After having done (macro (square X) (mul X X)) all occurrences of (square 3) will be translated to (mul 3 3) in future expressions read by newLISP.

The last statement happens in the rewrite function of the macro.lsp module:

Code: Select all

(set-ref-all pattern expr (eval $it) match))
Pattern is (square 3), evaluating it in (eval $it) returns the expansion replacement (mul 3 3).

Code: Select all

(define (test) (square 3)) => (lambda () (mul 3 3))

test => (lambda () (mul 3 3))
In both cases the macro facility does template expansion to generate the target LISP code.

PS: note that the following square macro is more efficient when a complex expression is past in X: (macro (square X) (pow X 2) ). It avoids the double evaluation of an expression in X as in (mul X X). In common Lisp the compiler would take care of this doing "common subexpression elimination". When we do macro expansion in a scripting language, we have to be sensitive to efficiency issues ourselves.

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Post by cormullion »

Cool. Look forward to playing with it. I'm happy for you to call it what you like, Lutz! :)

I'd like to know more about what these are really suitable for. I read the page you linked to first, and I get the general idea, but it did go on to say
Both Square and Square-Sum are inappropriate uses of macros.
There's a performance hit too, I suppose. What are people going to do with this...?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Macros with macro.lsp are good, where ever you would want to write a small function, but are afraid of wasting function-call overhead using 'define-macro'. Macros with macros.lsp are efficient but shorter to read and more descriptive in the program text:

Code: Select all

newLISP v.10.1.6 on OSX IPv4 UTF-8, execute 'newlisp -h' for more info.

> (module "macro.lsp")
MAIN
> (set 'x 2)
2
> (time (pow x 3) 1000000)
197.213

> (define (cube-a x) (pow x 3))
(lambda (x) (pow x 3))
> (time (cube-a x) 1000000)
381.352

> (macro (cube-b X) (pow X 3))
(lambda-macro (X) (expand '(pow X 3)))
> (time (cube-b x) 1000000)
191.922
> 
the macros takes the same time as the original, because internally it is doing the same. Internally its doing (time (pow x 3) 1000000) because (cube-b x) has been transformed.

Jeff
Posts: 604
Joined: Sat Apr 07, 2007 2:23 pm
Location: Ohio
Contact:

Post by Jeff »

This nice thing about these macros is that you are actually performing a compile-time transformation on the code. You can use them to easily define destructive setters on OOP classes, for example, because they would translate into the actual code that returns a reference. Perhaps something like this, using an assoc list to store your member vars in an instance variable:

Code: Select all

(macro (slot-value inst slot val)
  (if (nil? val) val (setf (assoc slot inst) val)))
Jeff
=====
Old programmers don't die. They just parse on...

Artful code

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Yes, exactly, good example how readability gets improved.

But note that the macro has to be written with uppercase variable names so 'expand' can work with it internally:

Code: Select all

(macro (slot-value Inst Slot Val)
  (if (nil? Val) Val (setf (assoc Slot Inst) Val)))
so when you code does:

Code: Select all

(slot-value my-inst the-slot my-val)
it will be rewritten to:

Code: Select all

(if (nil? my-val) my-val (setf (assoc the-slot my-inst) my-val)

As 'my-val' occurs 3 times in this macro and it could be that a caller passes some complex expression (blah (foo (x)), it may be better to define:

Code: Select all

(macro (slot-value Inst Slot Val)
  (let (value Val) (if (nil? value) value (setf (assoc Slot Inst) value))))
this way Val -> (blah (foo (x)) is only evaluated once.

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Post by TedWalther »

Lutz wrote:Fexprs were in newLISP from the beginning (v.1.0 was released in 1993). I thought that fexprs could do almost anything expansion/rewrite macros could do: creating special forms with non-standard evaluation rules. This is the reason why the name 'define-macro' has 'macro' in it. Its really a function, but its purpose is to let you do, what you normally do with macros.


But rewrite macros are coming to newLISP too:

Using a new 'reader-event' function in newLISP v.10.1.6, rewrite macros can be created. A rewrite function in an ew module macros.lsp packaged with v.10.1.6 intercepts code translation allowing for code tranformations during load of a program.

For details see here: http://www.newlisp.org/macros/
Hi Lutz, interesting module. Any chance this will be extended to work on character streams as well as well-formed list expressions?

Kazimir Majorinc
Posts: 388
Joined: Thu May 08, 2008 1:24 am
Location: Croatia
Contact:

Post by Kazimir Majorinc »

Yes, but other Lisp macros can do more complex things, for example

Code: Select all

(defmacro Square (X)
   '(* ,X ,X)) 

vs

(defmacro setq2 (v1 v2 e)
   (let ((e1 (comtran e)))
      (list 'progn (list 'setq v1 e1) (list 'setq v2 e1))))
There is no top apostrophe in the second example. If I understand well, your current macros cannot do that, that's why these are more C-like than CL-like. But it is not really hard to reach or improve over CL- macros. I defined one toy-example here:

http://kazimirmajorinc.blogspot.com/200 ... ation.html

Code: Select all

(load "http://www.instprog.com/Instprog.default-library.lsp")

(set 'x 3)
(println (time (pow x 3) 1000000)) ;=> 304

(define (cube-a x)(pow x 3))
(println (time (cube-a x 3) 1000000)); => 575

(define-macro (cube-b y) 
              'prepare-time    ; this means that it will be 
              (list 'pow y 3)) ; evaluated in "prepare time"
              
(println (prepare '(cube-b (cube-b x)))) ;=> (pow (pow x 3) 3)
(println (eval (prepare '(cube-b x)))); => 27
(println (eval (prepare '(time (cube-b x) 1000000)))) ;=> 317

(exit)
There is no reader-hook, so this eval - prepare - ' is necessary, but as said, it is only toy example.

You can have many kind of macros. You already have (1) current fexpr-like macros, (2) C-like macros you are demonstrating here, (3) CL-like-macros as in my library, some Scheme-like macros in future (4) ... clever naming strategy might be important now.

Jeff
Posts: 604
Joined: Sat Apr 07, 2007 2:23 pm
Location: Ohio
Contact:

Post by Jeff »

You know, Lutz, it would be nice if there were an operator that expanded at compile-time, rather than the entire function definition. That way, it could be evaluated inside a macro and provide the full power of both f-expressions and compile-time syntax transformations.
Jeff
=====
Old programmers don't die. They just parse on...

Artful code

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

TedWalther wrote:Any chance this will be extended to work on character streams as well as well-formed list expressions?
Jeff wrote:it would be nice if there were an operator that expanded at compile-time
Working the at the character stream level, rather than s-expression level would allow non-Lisp syntax, which for my taste is a bad idea. It also would have a much larger impact on newLISP's very fast source-code load time. Remember that in a scripting language everything has to happen during load/run-time. Initially I also explored the idea to expose the raw string stream to the user and let them do whatever they want to, using Unix an Awk like regular expressions API. But that would make the whole thing to impracticable, unless you are an regex artist

I think what we have now fits best into the general character of newLISP. Manipulation is limited to what well-formed lisp-expressions let us do, but the programmer who has chosen newLISP made that choice already.

'reader-event' is called with all top-level expressions. A top-level expression doesn't have to be parenthesized, it also can be single token, string or identifier. The current module macro.lsp limits to parenthesized expressions, but that does not need to be.
Kazimir Majorinc wrote:You can have many kind of macros. You already have (1) current fexpr-like macros, (2) C-like macros you are demonstrating here, (3) CL-like-macros as in my library, some Scheme-like macros in future (4) ... clever naming strategy might be important now.
Nobody has to take the current "macro.lsp" as religion, there are many other ways to define a macro function and use the new 'reader-event' hook. The current macro.lsp module is just the quickest, straight forward way to implement template-expansion macros and demo the basic principle of intercepting expressions and rewriting them. Other ways are only limited by one's imagination ... and coming up with enough confusing names for it :)

Jeff
Posts: 604
Joined: Sat Apr 07, 2007 2:23 pm
Location: Ohio
Contact:

Post by Jeff »

That's not what I meant. I meant that it would be nice to have the type of compile-time expansion you built using read-expr but in the context of a non-defining expression. More like letex than define, in other words. That way we could use them as part of a regular newlisp macro. Like the backtick in cl.
Jeff
=====
Old programmers don't die. They just parse on...

Artful code

Jeff
Posts: 604
Joined: Sat Apr 07, 2007 2:23 pm
Location: Ohio
Contact:

Post by Jeff »

Lutz, one other thing. Shouldn't this be reader-event, not read-event?

Code: Select all

(define (resume)
  (read-event rewrite))
Neither is documented in the manual.
Jeff
=====
Old programmers don't die. They just parse on...

Artful code

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

I corrected the typo here: http://www.newlisp.org/macros/macro.lsp.html

'reader-event' is part of the upcoming v.10.1.6. Here is the manual entry for it:

http://www.newlisp.org/downloads/develo ... ader-event

The "macro.lsp" is just one way to use 'reader-event'. Already the 'macro' function in macro.lsp isn't limited to transform defined functions. In (macro <pattern> <replacement>) any pattern could be there, it doesn't have to be a function call. Currently it is limited to a parenthesized expression, but you are free to change that ... and it doesn't have to be called 'macro' either, but made sence for what it currently does. It could be called 'compile' and write out a C program ;-)

'reader-event' just passes all top-level expressions and you 'reader-event' -handler can do any transformation on it you want.

Please nobody take the definition of 'macro' in macro.lsp as cast in stone, its just source which can be changed. The only fixed thing is the new built-in 'reader-event' function.

But these are the general rules you should follow when using 'reader-event':

1) Create a function, which somehow registers the patterns you want to translate and somehow tells what should be done with those patterns.

2) Create a transforming function (the reader-event handler) which searches each incoming expression for occurrence of this pattern and then does changes according do what has been prescribed.

3) Your event handler returns the new transformed expression to newLISP.

Jeff
Posts: 604
Joined: Sat Apr 07, 2007 2:23 pm
Location: Ohio
Contact:

Post by Jeff »

Thank you, Lutz. This is exactly the hook we need to extend the language. This is a good thing for the language.

Will reader-event be hooked into run-time compilation as well? That is, when I call read-expr, will the expression be first run through reader-event? I believe it should, so that translations that occur in one place happen everywhere. At the very least, it should be default. If it does not, it creates an inconsistency that must be accounted for in every program using read-expr and reader-event. It adds a workaround that must be memorized by new users, too.

It should not need to be hooked into load. Anything written to file should be in the expanded form in any case (at least, any code expressions written to file).
Jeff
=====
Old programmers don't die. They just parse on...

Artful code

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Post by Lutz »

Will reader-event be hooked into run-time compilation as well? That is, when I call read-expr, will the expression be first run through reader-event?
Yes, 'eval-string', 'read-expr' and 'load' or in other words everything in the system translating source code will pass it through 'reader-event'. That is the default. If 'reader-event' is specified it fundamentally changes and extends your language.

But you can suspend 'reader-event' at any moment doing (reader-event nil). Should probably add this to the documentation.

Of course there are also dangers to this, but that is nothing new. If in the past you redefined built-in functions before doing a 'load' or 'eval-string' you could shoot yourself in the foot as well, or you can use it to your advantage: see the profiling module somebody wrote redefining 'define'.

Using 'reader-event' you coud write a profiler, which is a little more sane by not redefining 'define' but injecting timer/counter code into the 'defined' function without touching 'define' itself.

Jeff
Posts: 604
Joined: Sat Apr 07, 2007 2:23 pm
Location: Ohio
Contact:

Post by Jeff »

In fact, I think *I* may have written that :)
Jeff
=====
Old programmers don't die. They just parse on...

Artful code

newdep
Posts: 2038
Joined: Mon Feb 23, 2004 7:40 pm
Location: Netherlands

Post by newdep »

Intresting!...
Even more stuff to study on ;-)
-- (define? (Cornflakes))

xytroxon
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm
Contact:

Post by xytroxon »

Thankfully, newLISP is NOT Common LISP ;p)

I think a clean macro module function that doesn't require an advanced CS degree to fathom how it mysteriously works is a definite plus for newLISP... The fact that its name would confuse LISPers is devilshly clever and delightful!

But again, what better name?

Google Search: macro

Macro - Wikipedia, the free encyclopedia
Look up macro or macro- in Wiktionary, the free dictionary. Macro is commonly known as the prefix in the word macorana, but may also be used in: ...
en.wikipedia.org/wiki/Macro - Cached - Similar

Google Search: macorana

Did you mean: macarena Top 2 results shown

Macarena (song) - Wikipedia, the free encyclopedia
Macarena is a Spanish song by Los del Río and Fangoria about a woman of the same name, or any woman from the La Macarena neighborhood of Seville, Spain. ...
Origin and history - Record breaking and worldwide ... - Music video
en.wikipedia.org/wiki/Macarena_(song) - Cached - Similar

Hum... macro... macarena... I think I got it!

Hey! macro-rena !

Where "rena" expands to reader event mnemonic algorithm

-- xytroxon

P.S. Can anyone play the macro-rena on their "mnemonica" for the shell game video audio? Ole!!!
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Post by TedWalther »

How about a macro like this?

Code: Select all

(macro function-to-detect-pattern function-to-transform-pattern character-stream)
Lutz, I understand you don't feel comfortable with people programming newLISP using the syntax of other languages. But the whole point of reader macros is that that syntax gets mapped onto newLISP syntax.

Take, for instance, the tk function. I'd like to mingle newLISP and tcl/tk calls at a deeper level than simply passing strings in various formats.

With read macros, quote and quasiquote and splice are easily implemented.

As with your current implementation of macro, reader macros will allow expansion to be done once as the expressions are read in, rather than being evaluated to generate new code every time, as is done with fexprs.

With a simple function pointer to the reader/parser, it can be arranged that there is NO performance slow-down when reader macros aren't enabled.

Ted
Lutz wrote:I corrected the typo here: http://www.newlisp.org/macros/macro.lsp.html

'reader-event' is part of the upcoming v.10.1.6. Here is the manual entry for it:

http://www.newlisp.org/downloads/develo ... ader-event

The "macro.lsp" is just one way to use 'reader-event'. Already the 'macro' function in macro.lsp isn't limited to transform defined functions. In (macro <pattern> <replacement>) any pattern could be there, it doesn't have to be a function call. Currently it is limited to a parenthesized expression, but you are free to change that ... and it doesn't have to be called 'macro' either, but made sence for what it currently does. It could be called 'compile' and write out a C program ;-)

'reader-event' just passes all top-level expressions and you 'reader-event' -handler can do any transformation on it you want.

Please nobody take the definition of 'macro' in macro.lsp as cast in stone, its just source which can be changed. The only fixed thing is the new built-in 'reader-event' function.

But these are the general rules you should follow when using 'reader-event':

1) Create a function, which somehow registers the patterns you want to translate and somehow tells what should be done with those patterns.

2) Create a transforming function (the reader-event handler) which searches each incoming expression for occurrence of this pattern and then does changes according do what has been prescribed.

3) Your event handler returns the new transformed expression to newLISP.

m i c h a e l
Posts: 394
Joined: Wed Apr 26, 2006 3:37 am
Location: Oregon, USA
Contact:

Re: Few questions ...

Post by m i c h a e l »

The new reader-event and macro module are muy bueno, Lutz. Thank you, once again, for all your hard work making newLISP such a fantastic programming language!

Unfortunately, it looks like we can't use the benefits of the new macro with dynamic dispatch:

Code: Select all

newLISP v.10.1.6 on OSX IPv4 UTF-8, execute 'newlisp -h' for more info.

> (module "macro.lsp")
MAIN
> (new Class 'Pair)
Pair
> (define (Pair:left p) (p 1))
(lambda (p) (p 1))
> (macro (Pair:right P) (P 2))
(lambda-macro (P) (expand '(P 2)))
> (set 'p (Pair "A" 99))
(Pair "A" 99)
> (time (:left p) 1000000)
1081.388
> (time (:right p) 1000000)
1876.25
> _
Yikes! It takes almost twice as long!

On the other hand, if we fully qualify the method, like so:

Code: Select all

> (time (Pair:left p) 1000000)
481.917
> (time (Pair:right p) 1000000)
54.432
> _
. . . we get the expected speed increases with the blazing speed of the new macro version.

Too bad, really. Accessors are a perfect candidate for these new C-like macros.

m i c h a e l

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Few questions ...

Post by Lutz »

Yes, because the macro facility looks for the exact pattern in the source code when specifying:

Code: Select all

(macro (Pair:right P) (P 2))
it will try to match (Pair:right *) not for (:right *)

On the other hand we cannot say:

Code: Select all

(macro (:right P) (P 2)) ; causes an error
Because the way the 'macro:macro' function in macro.lsp currently is defined, it looks for the first symbol in the left expression to be undefined, to avoid double definitions. Remember that (:right P) really is (: right P) and it will take the ':' as the functor and not care about the 'right'.

Even if we redefine the functions 'macro:marco' and 'macro:rewrite' in a more general way to take translate this kind of expressions, it still wouldn't work. Because the ':' colon operator extracts the correct class of the object only during run-time. When translating the code (:right p), there is no way to find out, that the class is 'Pair' without evaluating 'p'.

Locked