Returning copies of the values ...

[Kazimir Majorinc]

In one of my last blogs I noted that Newlisp, when using blocks of the expressions doesn't return the result of the evaluation of the last expression - but its copy.

It holds for begin, let, lambda, lambda-macro, silent ...

Why is that? Is there any possibility for changing it or additionally providing the versions returning actual value of the last evaluated expression, not its copy? Advantages would be some speed, but also some expressive power. I'm aware that passing actual values as arguments to functions breaks ORO, but returning actual values seems safe to me.

[Lutz]

In ORO each memory object is either referenced directly or indirectly by a symbol, if not, it must be stacked for later deletion. If we would return the original from (f 'L) when f => (fn (x) (eval x)), then the original content of L would eventually be destroyed by memory management.

The moment an intermediate result gets created during evaluation of a lambda or built-in function, we do not know how it will be used eventually. If it gets assigned to a symbol or gets part of a complex list structure referenced by a symbol, we would end up with 2 references to the same memory object if not copied. If copied but not assigned, it would linger in memory if not deleted.

[Kazimir Majorinc]

"The moment an intermediate result gets created during evaluation of a lambda or built-in function, we do not know how it will be used eventually. If it gets assigned to a symbol or gets part of a complex list structure referenced by a symbol, we would end up with 2 references to the same memory object if not copied. "

This is how I reason: we know we will not end up with two references, because you already designed the language not to break ORO if values of the symbols are used.

For example, take a look at push:

(push L1 some-list)

Not the value of L1, but its copy will be pushed. You did it as a guard that prevents breaking ORO. I guess you did similar thing, making copies everywhere where use of the original value treat to ORO. But on the places where it doesn't matter, you didn't. In that push expression, actual value of some-list, not its copy is changed. Because it is safe - value of some-list is not assigned to anything. So, you already took care about it.

The mechanism that prevents breaking ORO in the case value of some symbol is used should, I believe, prevent from breaking it in the case actual value of the last expression in the function is returned. So, what you actually have is double guards. For example,

(set 'f (lambda(x)(eval x)))
(push (f 'L) some-list)

What is pushed to some-list? Not actual value of L, because f returns the copy of its value. And not even the copy of value of L, but copy of the copy, because push will make the copy on its own. So, it appears to me that one copy is redundant - and that it is same in all situations when returning actual value from function can cause the problem, i.e, you already have guard against that problem.

But I really do not know much about Newlisp.
If I'm wrong, can you write an example such that returning of the actual value from function can break ORO?

[Lutz]

The memory-object producing function has no knowledge what will later happen to the object, if it will get consumed or not. For that reason it has to be stacked for deletion.

The consuming function has no quick way to take an object from the deletion stack for consumption and doesn't even know if its on there, because some places in the system are already optimized. You would have to look for the object throughout the stack, then take it out, if its there, which even increases time when dealing with small objects. For big objects we have reference passing with namespaces now for all functions (see 9.4.0 release notes) .

Memory management in newLISP has been thoroughly profiled and optimized over the years. There may be still places to improve, but all obvious ideas have been investigated already.

[Kazimir Majorinc]

OK, I think this issue can be divided on two independent, the first is efficiency issue, i.e. copy of the copy, how to return last expression without copy etc. You are probably right that it requires significant change of interpreter's internal data structures; and that it can really make interpretation slower in all cases except when large lists are copied.

However, expressive power is another issue. In Newlisp, we cannot write macro f such that following expression

(push 44 (f L))

calls f, which does something with L, and returns L so 44 is pushed into value of L. We can, however, write the macro f so it returns symbol L, and then

(push 44 (eval (f L))

So, we have to use one extra eval. I think that extra eval could be avoided -- without extensive changes of internal data structures of the Newlisp interpreter -- by addition of few new syntactical constructs, for example lambda/eval and lambda-macro/eval forcing such evaluation in caller environment.

Interpreter should, if it come across the

((lambda-macro/eval ...) expr1 ... )

evaluate it like it is:

(eval ((lambda-macro ...) expr1 ... ))

And that's all. Syntax can be different, not necessarily one keyword either.

Is it possible? If it is, I could explain why I think getting rid of that extra eval is important - but I do not want to make this post too long.

[Lutz]

Check also out the just uploaded 9.4.1 filling some gaps in reference passing. Now you can do:

Code: Select all

(define (f x) x)

(set 'L:L '(33 22 11))

(push 44 (f L))

L:L => (44 33 22 11)

ps: internally the evaluation rule looks like this: if a list or string parameter is expected but the argument evaluates to a namespace/context instead of a list or string then assume that the default functor of that namespace/context contains the list or string. Now you can use any function for either value or reference passing depending on the type of argument passed.

[Kazimir Majorinc]

Great, it is real "return by reference." Heavy artillery.

[Kazimir Majorinc]

However, these are the reasons for lambda/eval and lambda-macro/eval. If they can be easily implemented, they could provide some extra power, not needed for default functors, but still useful:

• [1] it's just handy to have one eval less for a general use. In Newlisp, difference between functions and macros are much smaller than in CL & Scheme - because of almighty eval you have and CL & Scheme have not. That's why they NEED macros, and for Newlisp, it is only syntactic sugar. Nevertheless, if sparing quotes is justified reason for macros - and I think it is - then sparing eval is justified as well.
  
  [2] it is particularly important for libraries - authors of the library want that functions are applicable as generally and as simple as possible, and instructing users to add extra eval is not practical. If author of the library decides to stay away of such "use eval!" instruction, additions of lambda/eval and lambda-macro/eval provide significant additional power.
  
  [3] lambda-macro/eval particularly is very similar to CL & Scheme macros: they evaluate result of the macro call in the caller environment. It is well tested concept, and it has some subjective and social value.

But as said, it holds only if it is easy to implement these two as it looks on the first sight.

;-----------------------------------------

Resume of Lisp functions (existing, abandoned and proposed):

Newlisp lambda ~ Lisp lambda
arguments -> evaluated -> body -> result -> not evaluated

Newlisp lambda-macro ~ ancientLISP fexpr
arguments -> not evaluated -> body -> result -> not evaluated

proposed lambda-macro/eval ~ Lisp macro
arguments ->not evaluated ->body -> result -> evaluated

proposed lambda/eval ~ no known equivalents
arguments -> evaluated -> body -> result -> evaluated