Request for a $this variable

Pondering the philosophy behind the language
Locked
TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Request for a $this variable

Post by TedWalther »

Lutz, $0-$15 are useful. I just learnt about $idx today. And $it is also useful. Can you please add $this to refer to the current context?

One idiom I use a lot is to make an object the way I want it, then use (new) to duplicate it. Having a $this variable will make it easier to call methods between contexts which then respond asynchronously by invoking another method in the calling context.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Request for a $this variable

Post by Lutz »

you can use:

Code: Select all

(context)
to get the current context. FOOP uses this in the default constructor:

Code: Select all

(define (Class:Class) 
    (cons (context) (args)))

(new Class 'MyClass)
See also here:
http://www.newlisp.org/downloads/newlis ... em_symbols

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: Request for a $this variable

Post by TedWalther »

Thanks
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

m35
Posts: 171
Joined: Wed Feb 14, 2007 12:54 pm
Location: Carifornia

Re: Request for a $this variable

Post by m35 »

Might there be some value in making a $this refer to the current lambda expression being executed? Would that make self-modifying functions more portable?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Request for a $this variable

Post by Lutz »

I have experimented with this quite a bit. newLISP internally keeps a lambda stack for error reporting. That makes it easy to access the current lambda.

In the end I left this idea alone because I couldn't come up with really usable use-cases. The only interesting usages are those where the current lambda expression was invoked by a symbol functor. But in those cases the lambda function can be accessed easily using the function name and a 'this' would not be required:

Code: Select all

> (define (foo x) (inc (last foo) x) 0)
(lambda (x) (inc (last foo) x) 0)
> (foo 1)
1
> (foo 1)
2
> (foo 10)
12
> 
All kinds of variations of above example have been shown over the years. The most interesting beeing Kazimir's code of a crawler-tractor, which runs eternally without using iteration or recursion:

Code: Select all

(define (crawler-tractor)
    (begin (println "Hi for the " (inc counter) ". time. ")
           (push (last crawler-tractor) crawler-tractor -1)
           (when (> (length crawler-tractor) 3)
                 (pop crawler-tractor 1)))) 
All other cases, where the lambda is anonymous and a 'this' would be required, are really not that useful, or the task they accomplish can easier be solved using conventional code.

I believe that functions with state are best realized using namespaces, as discussed today in another thread and as shown in the docs as the method of choice for functions with state in newLISP:

Code: Select all

(define (gen:gen)
    (inc gen:sum))

(gen) => 1
(gen) => 2

All these examples can also be found here: http://www.newlisp.org/index.cgi?Closures

Ps: note that it is only possible to modify a lambda expression this way. Reassigning the function inside its body would crash the system:

Code: Select all

(define (foo) (define (foo) "hello"))

(foo) => crash

Kazimir Majorinc
Posts: 388
Joined: Thu May 08, 2008 1:24 am
Location: Croatia
Contact:

Re: Request for a $this variable

Post by Kazimir Majorinc »

Really? I thought "$this" referencing the function is impossible, because of "one reference only."

I could think about two arguments for $this.

* anonymous functions changing themselves to store the state do not need to be explicitly deleted.
* the lists of the functions able to analyse or change themselves. Because of ORO these functions should reference themselves through their address in list, i.e. (L 1), (L 17) etc. It would be very fragile if someone pushes something in the list. Advantage of $this would be significant.

From practical point of view, self mutating functions can be used for similar purposes as closures. People seems to prefer contexts for such purposes, but that is another possibility.

m35
Posts: 171
Joined: Wed Feb 14, 2007 12:54 pm
Location: Carifornia

Re: Request for a $this variable

Post by m35 »

Hmm. Using self-modifying functions as closures might be pretty cool. You could create as many clones of those closures, each with their own state.

A very very sloppy example that could make use of the $this variable.

Code: Select all

(define (truck-class func)
	(let ((speed 100)
		(color "red"))
		(case func
			("set speed" (setf 
				(lookup 'speed (nth '(1 1) truck-class)) 
				(args 0) 
			))
			("get speed" speed)
			("set color" 
				(setf (lookup 'color (nth '(1 1) truck-class)) 
				(args 0) 
			))
			("get color" color)
		)
	)
)

(setq truck-instance truck-class)
(truck-instance "get color")         ; -> "red"
(truck-instance "set color" "blue")
(truck-instance "get color")         ; -> "red" (should be "blue")
(truck-class "get color")            ; -> "blue" (modified the original)

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Request for a $this variable

Post by Lutz »

Really? I thought "$this" referencing the function is impossible, because of "one reference only."
That is correct, and the way I had implemented this experimentally, was not with a variable $this but a function (this) returning a reference, so you could do:

Code: Select all

(define (foo x) (push x (this) -1)))
(this) would always return a lambda expression, either referenced by 'foo' or an anonymous copy, if lambda was anonymous.
self mutating functions can be used for similar purposes as closures
Yes, and all patterns known so far are discussed here (among other stuff):

http://www.newlisp.org/index.cgi?Closures

Kazimir's crawler-tractor was added to the closure page yesterday
You could create as many clones
m35's example makes a good case for a (this): when copying lambda expressions. But again: I think the code is too complex for what it achieves. It is also not very efficient (for an interpreted language), because method-dispatch is done wit a case statement.

Nevertheless it is an interesting approach and the last I want to do, is discourage further experimentation.

My current take on all of this is:

State is better captured using namespaces when dealing with object rarely deleted (because its not very efficient and manual).

For volatile objects, which come and go in bigger numbers FOOP is a better solution. FOOP keeps data and methods separate. Methods go into a class-namespace and instance-objects point to the class with their first list member. FOOP objects can be anonymous and are memory managed automatically.

Perhaps, as long as we try to use self-modifying functions to reproduce classic OO we are on the wrong track. newLISP is just not designed to be an classic OO language. The namespace system was primarily designed to facilitate grouping functions into modules, and this is how FOOP uses namespaces. Instead of trying to press newLISP into the traditional OO paradigm, perhaps we should try to come up with new ideas / paradigms. So far Michael's FOOP is the most efficient and elegant I have seen.

itistoday
Posts: 429
Joined: Sun Dec 02, 2007 5:10 pm
Contact:

Re: Request for a $this variable

Post by itistoday »

The problem with contexts though is that they're global.

This makes it very difficult to write modules (libraries) for newLISP.

In most other languages, for example, it is no problem to define a class called "Record" or "Person" or "Employee". And while this is possible in newLISP, you'd be a fool if you did, because contexts are global and can't be nested.

So instead, you must adopt some sort of naming convention. For example, for Dragonfly I'm creating artificial nested contexts by having a convention of prepending words to them. For example, all RESTful resources should have "Resource." prepended to their name, and all routes have "Route." prepended to their name.

If newLISP implemented real reference counting (which it can since it controls the '=' operator, setf), all sorts of things would be possible, you could then easily implement anonymous functions that have their own anonymous state, and you could pass them around without copying the entire state/context. If you didn't want to implement closures the way Scheme does, you could instead implement the function 'this' to create an anonymous context attached to the function itself:

Code: Select all

(define (counter) (fn () (inc (this x))))
(set 'cnt (counter))
(cnt) => 1
(cnt) => 2
'(this x)' would create (if it didn't exist) an anonymous context attached to the anonymous function containing the symbol 'x', and return a reference to x's data. You could do so many things with that, like create a real object oriented library where state is modified immediately, without creating an entire copy of all of the object's members. You would also obviously have anonymous closures in newLISP, and many of the name-conflict worries would go away.
Get your Objective newLISP groove on.

m35
Posts: 171
Joined: Wed Feb 14, 2007 12:54 pm
Location: Carifornia

Re: Request for a $this variable

Post by m35 »

I'm really impressed how newLISP's very simple memory management system (ORO: one-reference-only) has worked so well for so long.

Its simplicity makes its execution very predictable. You don't have to deal with a garbage collector arbitrarily pausing your scripts, and you don't have to worry about circular reference counting memory leaks. I'm sure it also keeps the executable smaller and easier to maintain and understand. It's also somewhat surprising that essentially no performance issues have come from the simple model. Lutz even managed to optimize much of it in the new v10.

For everything that newLISP can do, the ORO model has rarely been a problem, and then contexts can handle many of those problematic cases.

A new memory management system would be a huge change to newLISP. I would vouch for keeping ORO, if nothing else, just to be a real-life example that GC may not always be the ultimate solution for memory handling.


I would however like to see some changes made to context handling. As itistoday has shown, it really seems to have limitations.

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Re: Request for a $this variable

Post by cormullion »

itistoday wrote:contexts are global and can't be nested.
I remember a similar discussion some months ago. I don't remember Lutz saying that any changes to the context system were on the cards...

itistoday
Posts: 429
Joined: Sun Dec 02, 2007 5:10 pm
Contact:

Re: Request for a $this variable

Post by itistoday »

It's also somewhat surprising that essentially no performance issues have come from the simple model. Lutz even managed to optimize much of it in the new v10.
I agree what Lutz has done is great, but I see some fundamental issues with ORO, and one of them, although I could be wrong on this, is performance. I find it very hard to believe that copying lots and lots of lists all over the place, all the while allocating and deallocating that memory, is more efficient than simply incrementing or decrementing a register and testing it for 0.
m35 wrote:I'm really impressed how newLISP's very simple memory management system (ORO: one-reference-only) has worked so well for so long.

Its simplicity makes its execution very predictable. You don't have to deal with a garbage collector arbitrarily pausing your scripts, and you don't have to worry about circular reference counting memory leaks. I'm sure it also keeps the executable smaller and easier to maintain and understand.
When I say reference counting, I don't mean garbage collection, and in the scheme I'm advocating I think we could avoid the circular reference issue. Please let me know if I'm speaking gobble-de-gook (as I very well may be, I haven't studied this issue in depth), but here is what I mean:

In other languages, such as C, the '=' operator simply assigns values to registers, nothing more.

1) newLISP, and other interpreted languages, have the capability of overriding what '=' does.
2) newLISP could also, similar to Clojure, separate values from their symbols.

I've taken some time to think all this through, and I've provided my full thoughts on the issue (as well as how it could actually be done) here:

"RFC: A reference counted newLISP (and how to implement it)."
http://newlispfanclub.alh.net/forum/vie ... f=8&t=3151
Get your Objective newLISP groove on.

Kazimir Majorinc
Posts: 388
Joined: Thu May 08, 2008 1:24 am
Location: Croatia
Contact:

Re: Request for a $this variable

Post by Kazimir Majorinc »

Newlisp with ORO is similar to assembler. Just like in assembler, one object cannot be stored on two memory addresses, in Newlisp, one list or function cannot be the value of two different symbols. In assembler, if one wants the same object on two addresses, he has to copy. Since I realized that, I think about Newlisp symbol as about humanized, but also generalized memory address. (In assembler, one cannot delete memory address or insert new between two, in Newlisp one can do that with symbols.) So, how assembler programmers deal with problems that occur here, like excessive copying? They use indirection, not just addresses, but addresses to addresses. If list is stored in memory at address 100, assembler programmer can store number 100 into cells with addresses 500 and 600 and now 500 and 600 "indirectly" contain list. The problem is solved. Programming is bit more complicated, but that's it. Same indirection need to be used in Newlisp, and problem is solved in principle.

(set 'private-1889 (lambda()...))
(set 'f 'private-1889)
(set 'g 'private-1889)


Now, f and g indirectly evaluate to same function, and there was no copies. Of course, it is now "indirect" so one always must use that extra level of indirection, write (push 3 (eval f)) etc, but it is not essential problem. It is like pointers in C.

In real life, we automatically handle such indirections. I give you telephone number of pretty girl that want to meet you, and when you call that number, the voice tells that she has new phone number. If it happens, you do not simply proclaim ERROR: WRONG NUMBER and give up, you follow that new number. Newlisp doesn't do that:

> (setf mysin 'sin)
sin
> (mysin 3)

ERR: invalid function : (mysin 3)
>


It could - because that change only damages previous error behaviour. And it actually does that - with contexts:

> (set' MYSIN:MYSIN sin)
sin <410DE0>
> (MYSIN 3)
0.1411200081
>


Here Newlisp wasn't insulted with context, but it automatically looked into its default functor. Contexts are limited on the way they do not allow further nesting, but with symbols, one can use as much indirection as he needs. He losses ORO and GC didn't existed, so he is turned back to manual memory management, like in C. Not that bad. I'll stop here.

Itistoday: it was not the answer on your proposal, I just think loudly about nature of the ORO.

--
ORO by Tocak, the rock version of Macedonian folk dance music.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Request for a $this variable

Post by Lutz »

Misconceptions about ORO memory management:

- Memory gets allocated and deallocated all the time, and much cell copying is going on.

Not al all. ORO (One Reference Only) preallocates cell heaps and keeps a free-list from which cells are recycled and reused. Most of the time, copying or deleting does not mean allocating or freeing memory.

ORO is much about deciding what happens to a new created Lisp object in the future. From investigations other researchers did for Lisp and other languages we know, that most newly created objects can go away soon.

Traditional memory management methods manage much more memory objects than ORO, which schedules many for deletion right away, even for a slightly increased cost of copying. ORO does this with its result-stack; it holds result objects marked for deletion (recycling) in the future.

- ORO management is similar to C.

Far away from it. The result-stack in ORO is not a C stack. The C stack is used do push parameters and pop results after subroutine calls. ORO passes parameters using variables symbols and uses a symbol environment stack to keep track of symbol values up the calling hierarchy.

Also, in C memory is not managed automatically, if you copy a string, you have to allocate memory for the copy then do the copy. If this was just an intermediate result in a complex expression, you have to free that memory yourself.

- ORO is new in newLISP and unproven.

ORO was developed over the years and started out on two language projects before newLISP. The first was a string manipulating language similar to Tcl or Snobol, called DO, the second was a Prolog dialect called RuleTalk. Read more about it at http://www.donlucio.net on the projects page. DO was used in more then 150 companies in the early 80's to implement accounting and other IT software on early micro computers. RuleTalk was sold commercially and used mostly by European AI researchers on the DOS/MS-Windows PC.

- Automatic memory management follows just one method or the other.

Not at all. Most languages use a mixture of different techniques. There are many flavors of garbage collection. Many years, newLISP also had a mark and sweep garbage collector working under certain conditions. It was used less and less over the years removed completely only recently in v. 10.1.3.

Over time ORO has gotten less pure. In the beginning every memory object was treated the same way and only 'set' was optimized. Today all destructive built-ins have the same optimization which avoids to copying and pushing the result stack.

itistoday
Posts: 429
Joined: Sun Dec 02, 2007 5:10 pm
Contact:

Re: Request for a $this variable

Post by itistoday »

Thanks Lutz for clarifying some of those misconceptions about ORO!
Lutz wrote:Misconceptions about ORO memory management:

- Memory gets allocated and deallocated all the time, and much cell copying is going on.

Not al all. ORO (One Reference Only) preallocates cell heaps and keeps a free-list from which cells are recycled and reused. Most of the time, copying or deleting does not mean allocating or freeing memory.
That's good to know that it doesn't allocate/free memory each time, but isn't copying data still a possibly significant overhead that could be avoided through reference counting?

I readily believe that it's possible to make ORO efficient in certain ways, but my main gripe with it is not even efficiency, but rather how it limits certain coding possibilities, or provides difficult and unclean workarounds to problems that could be easily solved if everything was passed by reference. It just so happens that passing things by reference is a very efficient way to do stuff, and while there might be some overhead in other areas to implement such a scheme, just as you've implemented optimization strategies for ORO, I'm sure strategies could be implemented for the sort of reference counting I'm referring to. It could result in an even faster language, and it would certainly result in more code possibilities.
Get your Objective newLISP groove on.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Request for a $this variable

Post by Lutz »

… but my main gripe with it is not even efficiency, but rather how it limits certain coding possibilities …
Putting efficiency issues aside, the reason you would want reference a data object directly is to modify it. If there was no need to modify data there wouldn't be a need for referencing.

From a program hygienic point of view: the more code you have in your program modifying data objects the more error-prone gets the code. This is why people have tried to to design languages completely free of mutable data objects. Change is described in a pure functional way.

The functional approach avoids modification of data objects and state. It sees modifying data as a necessary evil, you cannot live without to complete your programming task, but you try to avoid modifying data, where possible. The main reason that destructive functions where introduced in newLISP was for efficiency. Ideally 'set/setf/setq' would be the only destructive operations saving the final result of a computation.

Referencing data directly is a necessity only for modifying data but should be kept to a minimum, to make a program less error-prone. When you must use referencing for the sake of efficiency, e.g. passing big data objects to user-defined functions, newLISP offers the method of passing data contained in a default functor. For complex structured data objects context handles can be passed.

itistoday
Posts: 429
Joined: Sun Dec 02, 2007 5:10 pm
Contact:

Re: Request for a $this variable

Post by itistoday »

Lutz wrote:
… but my main gripe with it is not even efficiency, but rather how it limits certain coding possibilities …
Putting efficiency issues aside, the reason you would want reference a data object directly is to modify it. If there was no need to modify data there wouldn't be a need for referencing.

From a program hygienic point of view: the more code you have in your program modifying data objects the more error-prone gets the code. This is why people have tried to to design languages completely free of mutable data objects. Change is described in a pure functional way.

The functional approach avoids modification of data objects and state. It sees modifying data as a necessary evil, you cannot live without to complete your programming task, but you try to avoid modifying data, where possible.
I disagree, at least with regards to newLISP.

Code becomes error-prone with regards to modification when there are concurrency issues at stake, i.e. when it's possible that your data could be modified in a dangerous way. This does not happen in newLISP where everything is in the same thread.

Your statements sound very much like the arguments the author of Clojure makes, but one of the main reasons Clojure was created was to address concurrency issues in a language where multi-threading is possible and therefore modifying data can lead to serious issues.

The way that Clojure went about solving this problem was very intelligent, it did not allow you to modify data, but it also did not simply make copies of data either. Instead it uses entirely new data structures at the core of the language that allow for "purely functional modification of data" without copying. When mutating these structures (lists, hash tables, arrays), instead of making copies of the entire thing it makes a new structure that *shares data* with the old structure, thereby avoiding copying the entire thing.

newLISP does not really have concurrency issues yet it will copy entire data structures anyway, and when it does so the modified data structures are new copies, not the crazy advanced shared-data ones that Clojure has.

I thought that newLISP was supposed to be a simple and practical language that did not have to worry about these issues and therefore could take full advantage of the improvement in readability and performance that you have when everything is done in a single thread.

newLISP doesn't need to worry about the multi-threaded issues that other languages worry about. It's certainly possible to introduce them using shared memory between two newLISP processes, but why would you do that? And even if you did you can always use standard synchronization mechanisms. The vast majority of tasks newLISP is used for can be solved in a single thread, or by using the new actor functions.

Another reason that is championed as a virtue of functional programming languages is side-effects. Where modification leads to problems is with unmentioned side-effects. This however is not a fault of a language for allowing direct modification of data, but rather the fault of poorly written APIs that do not state what they actually do.

Saying that newLISP should be handicapped because of any of these reasons is in my view unjustified, and again, limits the capabilities of the language.
The main reason that destructive functions where introduced in newLISP was for efficiency. Ideally 'set/setf/setq' would be the only destructive operations saving the final result of a computation.

Referencing data directly is a necessity only for modifying data but should be kept to a minimum, to make a program less error-prone. When you must use referencing for the sake of efficiency, e.g. passing big data objects to user-defined functions, newLISP offers the method of passing data contained in a default functor. For complex structured data objects context handles can be passed.
There are plenty of times when you would want easy reference passing. Any data structure needs this. Queues, n-ary trees, hash tables, priority queues, etc. All of these data structures need to be able to reference their data to be able to manipulate it efficiently. Writing these data structures in newLISP is neither efficient nor simple.

The use of contexts (a global namespace and grouping mechanism) as pointers is, in my view, a hack and nothing more. Pointers should be anonymous because their use is quite common. If in C, for example, every pointer suddenly became a global symbol, you would not be able to use the language for any practical purpose. Yet this is what newLISP has done. For me at least, the use of newLISP's default functors as pointers is an option to be used only in the rarest of situations, particularly when it comes to writing modules for newLISP, otherwise I'd be liable to mess up someone else's code.
Get your Objective newLISP groove on.

itistoday
Posts: 429
Joined: Sun Dec 02, 2007 5:10 pm
Contact:

Re: Request for a $this variable

Post by itistoday »

I'm willing to accept that perhaps newLISP just isn't designed to be used in certain ways, and that to do certain things you either don't do them at all, do them in C instead, or use default functors (albeit sparingly). But I am simply frustrated because I don't see why this has to be the case. I really like newLISP and I wish I could apply it to a larger set of problems, but either I can't or the solution isn't pretty.
Get your Objective newLISP groove on.

cormullion
Posts: 2038
Joined: Tue Nov 29, 2005 8:28 pm
Location: latiitude 50N longitude 3W
Contact:

Re: Request for a $this variable

Post by cormullion »

Although I'd hate to see you go, I can't help thinking but that you'd be happier with something like http://programming.nu/. It's got the Obj-C stuff you know about, plus (presumably) much object-oriented orientation. And some iPhone activity. They could do with a new logo, though.

itistoday
Posts: 429
Joined: Sun Dec 02, 2007 5:10 pm
Contact:

Re: Request for a $this variable

Post by itistoday »

cormullion wrote:Although I'd hate to see you go, I can't help thinking but that you'd be happier with something like http://programming.nu/. It's got the Obj-C stuff you know about, plus (presumably) much object-oriented orientation. And some iPhone activity. They could do with a new logo, though.
Thanks, it's good to know I haven't pissed everyone off with my ranting. ;-)

I did look into that a while ago actually and I decided I preferred newLISP. :-)

For iPhone/OS X type development I have no problem doing regular Objective-C (the non-garbage collected variety).
Get your Objective newLISP groove on.

Locked