Changes in newLISP v.10.0 (previous init.lsp)

[cormullion]

Can't remember if I asked this before...

It would be great if newLISP would load a local ~/.init.lsp file even if there's no /usr/share/newlisp/init.lsp. It seems that some newlisp installers (or re-installations) revert back to having just an 'init.lsp.example' file which doesn't get loaded, and therefore doesn't load any other init.lsp.

Or you could have the /usr/share/newlisp/init.lsp always exist, but contain only this code uncommented:

Code: Select all

(constant (global '$HOME) (or (env "HOME") (env "USERPROFILE") (env "DOCUMENT_ROOT")))
(if $HOME (catch (load (append $HOME "/.init.lsp")) 'error))

Part of the reason for this is that making changes to the /usr/share/newlisp directory requires authority... I can't make changes to this directory myself, on some systems (eg my nearlyfreespeech.net sites. So I can't run any init code.

[xytroxon]

Can you use the link.lsp util?

link.lsp
----
;; the new executable file will not load init.lsp but
;; the appended file instead
___

The appended file code could then look for and load your init.lsp...

-- xytroxon

[Lutz]

The next version of newLISP will try to load both: first the init.lsp in the install directory and second the home directory .init.lsp file.

[Jeff]

You should make it check the home directory first. If it finds it, it then stops. If it does not, it goes for the global, but only if there is nothing in the home directory.

[cormullion]

Either option would be great! Thanks for considering this, Lutz.

[Lutz]

It will be as in Jeff's proposal. Having your own $HOME/.init.lsp then lets you completely bypass system wide setting in case you don't want them. This will work in Windows too using the USER_PROFILE environment variable.

And here the big surprise: in the next generation of newLISP (version 10 and after) you can handle references returned from built-in functions:

Code: Select all

(set 'L '(a b c d e f g))

(setq (L 3) 'D) 
(setq (first L) 'A)

D => (A b c D e f g)

; or

(set 'A '((a 1) (b 2) (c 3)))

(setq (assoc 'b A) '(b 222))

A => ((a 1) (b 222) (c 3))

; or 

(push 4 (assoc 'c A) -1)

A => ((a 1) (b 222) (c 3 4))

; or

(set 'L '(w e v f b t))

(pop (sort L)) => b

L => (e f t v w)

; another one

(set 'L '((a 1) (b 1 (2 3) 4) (c 3)))

(push 99 (lookup 'b L -2))

L => ((a 1) (b 1 (99 2 3) 4) (c 3))

... etc ...

This type of handling list or array references is usual in most programming languages. Many newbies to newLISP are trying this intuitively at first.

Changes for this also significantly speed up handling of bigger lists and arrays, decrease memory usage and obsolete several of the destructive setter-functions now in use.

Most important its makes newLISP easier to learn and use.

[cormullion]

Cool! I think!

[Jeff]

Will that be released anytime soon? I like setf :)

[Kazimir Majorinc]

This looks great to me, especially set looks more abstract and easier to use.

Code: Select all

(set 'L '(w e v f b t))

(pop (sort L)) => b

L => (e f t v w)

It is certainly better - if one wants to "pop" copy of the sorted value of L, he can make the copy explicitly. How do you plan to accomplish that? Is it change in interpreter or specifically in functions sort or pop?

[Lutz]

Is it change in interpreter or specifically in functions sort or pop?

It is both, changes in newLISP's core routines processing expressions and changes in built-in functions. Its a type of optimization of ORO memory management, before made only for set, but now available system wide.

All built-in functions when returning lists or arrays bound to symbols or part of those lists or arrays, now return references instead of copies.

This affects the following functions:

assoc
first
last
lookup
nth
replace
reverse
rotate
set (returned reference already)
setf (new function)
setq (old function now working like setf)
set-ref
set-ref-all
sort
swap

and this also effect all control structures, which don't maintain local variables:

begin
if
if-not
cond
case
when
unless
while
until
do-while
do-until

this means you cannot use 'begin' any more to make destructive function non-destructive, you will now use something like (define (copy x) x) and then do: (sort (copy mylist)) to make the sort non-destructive.

Reference-returns together with the new 'setf' make it convenient to change lists and arrays:

(set 'lst '(a b c d))
(setf (last lst) 99)
lst => (a b c 99)

The normal old 'set' has not changed (it always returned a reference, but nobody was aware of it). 'setq' now points to 'setf' internally. I opted to keep both functions, to make it easier for converts from other Lisps. Many have 'setq' and 'setf' and want to keep on using them as they did before. It also improves readability and understandability of code having both.

Version 9.9.2 (a precursor to newLISP v.10.0) is ready, but I am still working on documentation changes. This version will be posted latest on Monday, perhaps earlier.

Although most old code is compatible with the new generation of newLISP, I opted of jumping from version 9 to 10, because the changes mark a new style of programming in newLISP which is closer to other programming languages and more intuitive.

As before user defined lambda function will return copies and copy parameters (except when using default functors).

[DrDave]

EDIT

Nevermind... I see that in the post above Lutz already sorted it out for me with the remarks about returning copies versus references.
======================================

I tried this as the separate opertions with v9.4.8

Code: Select all

(set 'L '(w e v f b t))
=> (w e v f b t)

(sort L)
=> (b e f t v w)

(pop L)
=> b

L
=> (e f t v w)

So both sort and pop were destructive as expected.

But combing thus:

Code: Select all

(set 'L '(w e v f b t))
=> (w e v f b t)

(pop (sort L))
=> b

L
=> (b e f t v w)

I see that the proposal will be for the the combined functions to give the same final list that the separate functions give. So, I am wondering why currently pop is destructive when used alone, but is not destructive when combined in an expression. Is it because sort did its thing and sent a copy of its result rather than its original for pop to work on?

[cormullion]

Although most old code is compatible with the new generation of newLISP,

I don't understand the changes yet (need to take my time) ... But for the executive summary - what code will stop working at version 10?

[Lutz]

In all of the code shipped with newLISP, from 55 lisp files, only 3 files needed minor edits. Applications newlisp-wiki and newlisp-ide on the newlisp.org web-site did not require any changes. Most of the changes where due to dropping 'set-nth', 'nth-set', 'set-assoc', 'assoc-set' in favor of the new 'setf' which can do them all in combination with 'nth' or 'assoc'.

A quick scan of "Introduction to newLISP" shows that there are basically only two areas of change, and I can help to correct the examples. The following is already about 80% of it.

'set-nth', and 'nth-set' are substituted by 'setf' in combination with nth:

Code: Select all

; the old
(set-nth 5 data 0)

; gets now
(setf (nth 5 data) 0)
; or
(setf (data 5) 0)

; the old
(nth-set 5 data "")

;gets now
(setf (nth 5 data) "")
; or
(setf (data 5) "")

There are two or three more like this.

'set-assoc' ans 'assoc-set' are subbstitued by 'setf' in combination with 'assoc':

Code: Select all

; the old
(assoc-set (table charge)  (list charge (+ result 1)))

; gets now
(setf (assoc charge table)  (list charge (+ result 1)))

; and this
(set-assoc (sol-sys "Jupiter") '("Jupiter" 11.2 ... 64))

; gets now
(setf (assoc "Jupiter" sol-sys) '("Jupiter" 11.2 ... 64))

May be and additonal short chapter about 'setf' or may be just use/introduce 'setf' in the part of the introduction where you explain how to modify lists and arrays. Those are the parts I corrected the examples above. The changes make the code faster in most instances and always save memory.

There are no changes required in your new time module.

I will go through the introduction at a later time again to see if some of the code could be written more elegantly and efficiently using reference-returns possible now. The fact that many built-in functions now return references will typically not disturb any previously written code, just give the programmer more expressiveness for future programs written.

There will also be a short "conversion guide" with the final 10.0 release. Most of it is already written on this post. But there is also beloved 'unless' reintroduced without the else clause, and thats pretty much all of it :-)

[cormullion]

Thanks for the attention to detail, Lutz... I'm not too worried about the tutorial document yet (only just finished a revision... :).

Did you really mean you are

dropping 'set-nth', 'nth-set', 'set-assoc', 'assoc-set'

or just providing improved functions and leaving the existing ones in? (I hope the latter - why do you have to drop them?)

Perhaps this important topic should be moved to a sticky topic...?

[Kazimir Majorinc]

Did you really mean you are

dropping 'set-nth', 'nth-set', 'set-assoc', 'assoc-set'

or just providing improved functions and leaving the existing ones in? (I hope the latter - why do you have to drop them?)

I believe he'll leave these for a decent time, however, your main problem (because you have lot of code) is that code like

(pop (sort L))

will silently change behaviour. So, you'll maybe have to "port" your programs if you want to use them with new versions of Newlisp. To replace "sort" with "copy-sort" you'll write etc. It might look ugly on the first sight, but it is not that bad really; back in 1970's almost never happened that two computers or two generations of compiler accepted same source code. Day or two of job maybe.

[Lutz]

I really mean dropping them. It has streamlined code for 'nth' and 'assoc' considerably, as the setter code was part of it. Dropping them made the remains of 'nth' and 'assoc' leaner and faster and there is less to learn. Keeping newLISP free of glut, fit and trim ;-)

Fortunately these four functions where used relatively rarely, so in the end the compatibility impact is not that big and we have a more unified approach using 'setf' for modifying lists and array by reference. Also more conform to what other programming languages do.

The 9.x.x series (really starting at v.8.9 two years ago) has been a constant explosion of new features. With version 10.0 it is time to consolidate and do some cleanup, where things have gone too messy in the language caused by rapid introduction of many new features (which was necessary).

ps: I changed the topic of the whole thread.

[cormullion]

Fortunately these four functions where used relatively rarely.

? True perhaps for assoc-set and set-assoc - they were relatively new - on my system only Jeff and Norman seem to have used it in their code. But nth-set (and to some extent set-nth) are more widely used.

Perhaps it's just me that's surprised - perhaps other programming languages are constantly dropping core functions and changing existing ones.

I'll shut up now! :)

[Lutz]

All programming languages go through sometimes major changes, and they all manage it differently.

On one extreme are languages, which never change, they get used, then eventually die out making place for new ones. Others change in major shifts, like Perl from version 3 to 4 to 5 to 6, or like Python 1 to 1.4 to 1.6 to 2 to upcoming 3.

newLISP always has followed a more evolutionary model in smaller steps, constantly changing but smaller shifts in-between versions. When you start converting your code e.g. identifying all 'nth-set' you will see its just not such a big deal and most of your code will keep running unchanged ;-)

What makes a language easier to use and learn and expressive is the degree to which words can be combined and work together in composites.

Reference returns and 'setf' do just that, allowing more composites between different built-in functions and in this way increasing the expressiveness of the language and the ease to learn it.

It also leads to more consistent and efficient code internally. Here an example:

Code: Select all

(set 'L '(a b c))

(time (setf (nth 0 L) 'z) 1000000) => 207 ; in version 9.9.2
(time (setf (L 0) 'z) 1000000)     => 149 ; in version 9.9.2
(time (setf (first L) 'z) 1000000) => 126 ; in version 9.9.2

(time (nth-set 0 L 'z) 1000000)    => 301 ; in version 9.4.8
(time (nth-set (L 0) 'z) 1000000)  => 194 ; in version 9.4.8

L => (z b c) ; same result for all versions

Execution times are much faster for the composite idioms in the new version. All five make the same change to L, but the first three work by combining built-ins to new composites.

Look for 9.9.2 tomorrow :-)

[cormullion]

My point has nothing to do with the changes - I welcome them (if there's a good reason for them)!

I'm more concerned with the transition process, which you seem anxious to minimize. It's important to be able to write code that runs on two or more releases. You're proposing dropping nth-set and introducing setf at the same release, aren't you? How could I write code that runs on either 9.4 or 10?

[Lutz]

Here is a snippet from stat.lsp module which is distributed with newLISP:

Code: Select all

(if (< (sys-info -2) 9902)
    (nth-set (ps 0) (div (first ps) 2))
    (setf (ps 0) (div (first ps) 2)))

I had to do this only twice but have all modules published on newlisp.org compatible for past and future version.

If you adhere strictly to the above syntax for 'nth-set' and you have many of those, you could do once at the beginning of old files:

Code: Select all

(if (>= (sys-info -2) 9902)
    (constant 'nth-set setf))

or at the beginning of new files you write:

Code: Select all

(if (< (sys-info -2) 9902)
    (constant 'setf nth-set))

Both, old and new files will than run on old and new newLISP.

In my own code I opted for the first solution, because I had very few places to change. If you have cases like the one Kazimir was referring to, like: (pop (sort lst)) and you relied on the pop to be nondestructive for lst, you would have to rewrite it to (first (sort lst)) to give the same result on both versions of newLISP. But those 'Kazimir' cases are very rarely occurring. I did not have even one in all of my code.

Ps: currently, Sept 14, http://www.newlisp.org is down, probably because of scheduled IP-renumbering at the hosting site. Same seems to be true for http://unbalanced-parentheses.nfshost.com

- update: both sites just came back

[Jeff]

Lutz,

When can we expect a dev release? Also, will we be able to create our own reference functions by returning a list reference expression? I.e.:

Code: Select all

(define (my-first lst) (expand '(lst 0) 'lst))