development release newLISP v.9.2.10

[Lutz]

New functions for search and replace in nested lists. A fix for trailing slashes in the directory predicate on Win32.

Lutz

for files and changes notes see: http://newlisp.org/downloads/development/

[HPW]

I think you meant:

http://newlisp.org/downloads/development/

(As always)
;-)

[Lutz]

thanks for catching this, corrected in the original post.

Lutz

[cormullion]

It's working well here (MacOS X 10.4.11 Intel). Cool new functions too!

[Cyril]

It's working well here (MacOS X 10.4.11 Intel). Cool new functions too!

I haven't looked into new functions yet, but two my scripts that I use on regular basis (shortcuts to them are even placed to my quick launch bar) are working fine. I do always test this after broken 9.2.6, but, thanks to Lutz, all of 9.2.7 - 9.2.10 was flawless.

[Lutz]

... glad 9.2.10 has been fine so far, this development release is feature complete for stable release 9.3, which is due beginning January 2008.

Lutz

[pjot]

This release compiles fine in Tru64 5.x but on 4.x I see the following (minor) issues:

Discovered Tru64 Unix, please read doc/TRU64BUILD
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET newlisp.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-symbol.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-math.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-list.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-liststr.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-string.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-filesys.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-sock.c
cc: Warning: nl-sock.c, line 407: In this statement, & before array "(dest_sin.sin_zero)" is ignored. (addrarray)
memset(&(dest_sin.sin_zero), '\0', sizeof(dest_sin.sin_zero));
---------------^
cc: Warning: nl-sock.c, line 407: In this statement, & before array "(dest_sin.sin_zero)" is ignored. (addrarray)
memset(&(dest_sin.sin_zero), '\0', sizeof(dest_sin.sin_zero));
---------------^
cc: Warning: nl-sock.c, line 861: In this statement, & before array "(dest_sin.sin_zero)" is ignored. (addrarray)
memset(&(dest_sin.sin_zero), '\0', 8);
-------^
cc: Warning: nl-sock.c, line 861: In this statement, & before array "(dest_sin.sin_zero)" is ignored. (addrarray)
memset(&(dest_sin.sin_zero), '\0', 8);
-------^
cc: Warning: nl-sock.c, line 1191: In this statement, & before array ""
"" is ignored. (addrarray)
if(newLine) write(handle, &LINE_FEED, strlen(LINE_FEED));
--------------------------^
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-import.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-xml.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-web.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-matrix.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-debug.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET pcre.c
cc newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o nl-import.o nl-xml.o nl-web.o nl-matrix.o nl-debug.o pcre.o -lm -lrt -ldb -lbsd -o newlisp
strip newlisp

Also running 'qa-dot' in Tru64Unix 5.x fails for two tests:

TESTING: newlisp FINISHED WITH ERRORS:

>>>> pretty-print failed nil
>>>> rand failed nil

total time: 6679

I will investigate this later this week.

Peter

[pjot]

The (rand) test fails because of this:

newLISP v.9.2.10 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (set 'sum 0)
0
> (dotimes (x 1000)(inc 'sum (rand 2)))
65385830
> (and (<sum> sum 400)(list? (rand 10 100))))
nil

The 'rand' function does not work:

> (rand 2)
33878

In the sourcecode of 'nl-math.c' this is going wrong:

Code: Select all

rnum = ((scale * random())/(RAND_MAX - 1));

From the manpage of 'random()':

The random() function uses a nonlinear additive feedback random number gen-
erator employing a default state array size of 31 integers to return suc-
cessive pseudo-random numbers in the range from 0 to (2^31)-1. The period
of this random number generator is approximately 16*((2^31)-1). The size
of the state array determines the period of the random number generator.
Increasing the state array size increases the period.

If I run this for example:

Code: Select all

scale * random()

Then the output is 286605828.000000 or something the like. So the 'random()' function already generates a huge number. For some reason, the function bails out here:

Code: Select all

rnum = ((scale * random())/(RAND_MAX - 1));

if(params->type == CELL_NIL)
        return(stuffInteger((UINT)rnum));

So the params->type == CELL_NIL and the huge number is returned.

Peter

[pjot]

Phew found the cause. The value of RAND_MAX is different in Linux and Tru64Unix.

Linux:
RAND_MAX = 2147483647

Tru64Unix:
RAND_MAX = 32767


Therefore the result of the calculation ((scale * random())/(RAND_MAX - 1)) is much higher. And the test for (rand) fails.

Peter

[Lutz]

- prettty-print

the default changed to 80 instead of 64 characters, and this has not been updated in the qa-dot and qa-comma files

- rand

had the last change in 9.2.6, this is fine MAc OS X when compiling as 64-bit version, so I don't know at the moment how to debug this for TRU64. I wonder, If I could have access to your machine?

Pehaps you insert the following code for me to debug:

Code: Select all

long testRand; /* for debugging */

testRand = random();
printf("scale=%f  testRand=%ld RAND_MAX=%ld  scale * testRand=%f\n", 
    scale, testRand, RANDOM_MAX, scale * testRand);

I may have gotten wrong some of the formatting characters for TRU64. The problem problem probably happens in the multiplication and implicit type conversions to float and back to int.

worst case we have to go back to pre 9.2.6 code for rand, let me know if 9.2.5 tests fine for rand on TRU64.

Lutz

ps: would be great to TRU64 working for 9.3 in January

[pjot]

Hi Lutz,

Thanks for the reply. This is the result of your code:

[peter@olga]# ./newlisp
newLISP v.9.2.10 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (rand 2)
scale=2.000000 testRand=2078917053 RAND_MAX=32767 scale * testRand=4157834106.000000
8747

Hope it helps...? Maybe the variable RAND_MAX must be redefined in 'newlisp.h'?

I do not have the sourcepackage for 9.2.5 anymore, but 9.2.0 shows:

newLISP v.9.2.0 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (rand 2)
1

So that looks good.

ps: would be great to TRU64 working for 9.3 in January

Yes! I hope so! :-)

Peter

[Lutz]

Thanks Peter I have redefined RAND_MAX 0x8FFFFFFF in newLISP.h conditionally for the TRU64 flavor.

Lutz

[pjot]

Thanks, please make sure the redefinition occurs AFTER the '#include <stdio>' statement.

Compilation becomes very ugly but it works correct now:

cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET newlisp.c
cc: Warning: newlisp.h, line 478: The redefinition of the macro "RAND_MAX" conflicts with a current definition because the replacement lists differ. The redefinition is now in effect. (macroredef)
#define RAND_MAX 0x8FFFFFFF
-----------------^
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-symbol.c
cc: Warning: newlisp.h, line 478: The redefinition of the macro "RAND_MAX" conflicts with a current definition because the replacement lists differ. The redefinition is now in effect. (macroredef)
#define RAND_MAX 0x8FFFFFFF
-----------------^
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-math.c
cc: Warning: newlisp.h, line 478: The redefinition of the macro "RAND_MAX" conflicts with a current definition because the replacement lists differ. The redefinition is now in effect. (macroredef)
#define RAND_MAX 0x8FFFFFFF
-----------------^
....

[peter@olga]# ./newlisp
newLISP v.9.2.10 64-bit on Tru64Unix, execute 'newlisp -h' for more info.

> (rand 2)
1
>
[/quote]

Cheers

Peter

[pjot]

In the headerfile 'stdlib.h' on Tru64Unix I see this:

Code: Select all

#ifdef _BSD
#define RAND_MAX        2147483647      /* max value returned by rand   */
#else
#define RAND_MAX        32767           /* max value returned by rand   */
#endif

Now, if I add -D_BSD to the makefile, and recompile, I get this:

[peter@olga]# make
./build

Discovered Tru64 Unix, please read doc/TRU64BUILD
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET -D_BSD newlisp.c
cc: Error: newlisp.c, line 69: This declaration contains a redefinition of "opsys". The previous declaration is at line number 440 in file newlisp.h. (redef)
int opsys = 9;
----^
cc: Error: newlisp.c, line 70: This declaration contains a redefinition of "ostype". The previous declaration is at line number 441 in file newlisp.h. (redef)
char ostype[]="Tru64Unix";
-----^
*** Exit 1
Stop.
*** Exit 1
Stop.

Sorry for all the spam...

[Lutz]

In the make file you can use only one definition for a flavor, if not the code may break in many more places.

I will put the define for MAX_RAND after all other includes in newlisp.h. But I think the real fix would be to fix somehow the error in the TRU64 header files, or live with the repeated warnings that MAX_RAND redefined.

Perhaps there is an option in the TRU64 compiler to suppress the macro-redefinition warning?

Lutz

[Lutz]

There are only 3 occurrences of RAND_MAX in the code base and all in nl-math.c. I will code around this introducing MY_RAND_MAX and set it conditionally in newlisp.h.

Lutz

[pjot]

Thanks, this works now! The (rand) function is surviving the test. One remarks though:

1) There is still a redefinition in 'newlisp.h' of RAND_MAX at line 99-101, can you remove it?

Thanks!

Peter

[pjot]

Lutz,

Now I found another issue but I also found the cause.

It seems that your change in (ends-with) crashes newlisp when calling this line from a program:

Code: Select all

(ends-with "newlisp" "LISP" nil)

However, running in interactive mode, this line work well. But running from a program with shebang "#!/usr/bin/newlisp", the program creates a coredump.

Hunting down this issue I arrive at this piece of code in 'nl-liststr.c':

Code: Select all

1312         /* append $ to the pattern for anchoring at the end */
1313         keydollar = alloca(klen + 4);
1314         *keydollar = '(';
1315         memcpy(keydollar + 1, key, klen);
1316         memcpy(keydollar + 1 + klen, ")$", klen);
1317         *(keydollar + klen + 3) = 0;
1318         klen = klen + 3;
1319     if((pos = searchBufferRegex(string, 0, keydollar, slen, options, &klen)) != -1)
1320                 {
1321                 if(pos + klen == slen)
1322                         return(trueCell);
1323                 }       
1324         }       

The program actually crashes when calling searchBufferRegex. The problem here is the usage of 'alloca'. From the Tru64Unix manpages:

[Tru64 UNIX] The alloca() function allocates the number of bytes of space
specified by the size parameter in the stack frame of the caller. This
space is automatically freed when the function that called the alloca()
function returns to its caller. The contents of the memory is undefined.

From the Linux manpages:

BUGS
The alloca() function is machine and compiler dependent. On many systems its implementation
is buggy. Its use is discouraged.

On many systems alloca() cannot be used inside the list of arguments of a function call,
because the stack space reserved by alloca() would appear on the stack in the middle of the
space for the function arguments.

If I change the 'alloca' for a plain 'malloc', everything runs fine and there is no crash. But the memory must be freed afterwards, of course.

Peter

[Lutz]

no problem I can change to malloc() for TRU64

Lutz

[pjot]

Maybe you have to also change it for Linux, please check the manpage for 'alloca'...

See my post above, I just changed it.

Thx
Peter

[Lutz]

seems to be TRU64 specific. alloca() in 'ends-with' is fine on Windows, Linux BSDs and Mac OS X 64bit.

BTW there are 26 places in the newLISP code base where alloca() is used, but TRU64 seems to behave fine on those. alloca() is allocating memory from the call stack.

There is a new version in your space ;-)

Lutz

[pjot]

Well, those are convincing arguments... :-)

This latest release is running as a dream:

[peter@olga]# make
./build

Discovered Tru64 Unix, please read doc/TRU64BUILD
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET newlisp.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-symbol.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-math.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-list.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-liststr.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-string.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-filesys.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-sock.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-import.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-xml.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-web.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-matrix.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET nl-debug.c
cc -ieee -pedantic -c -O3 -DSOLARIS -DTRU64 -DNEWLISP64 -D_POSIX_PII_SOCKET pcre.c
cc newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o nl-import.o nl-xml.o nl-web.o nl-matrix.o nl-debug.o pcre.o -lm -lrt -ldb -lbsd -o newlisp
strip newlisp
[peter@olga]#
[peter@olga]# ./qa-dot

Testing built-in functions ...

! != $ % & * + - / : < << <= = > >= >> NaN? ^ abs acos acosh add address amb and append append-file apply args array array-list array? asin asinh assoc atan atan2 atanh atom? base64-dec base64-enc bayes-query bayes-train begin beta betai bind binomial callback case catch ceil change-dir char chop clean close command-line cond cons constant context context? copy-file cos cosh count cpymem crc32 crit-chi2 crit-z current-line curry date date-value debug dec def-new default define define-macro delete delete-file delete-url destroy det device difference directory directory? div do-until do-while doargs dolist dostring dotimes dotree dump dup empty? encrypt ends-with env erf error-event error-number error-text eval eval-string exec exists exit exp expand explode factor fft file-info file? filter find find-all first flat float float? floor flt for for-all fork format fv gammai gammaln gcd get-char get-float get-int get-long get-string get-url global global? if ifft import inc index int integer integer? intersect invert irr join lambda? last legal? length let letex letn list list? load local log lookup lower-case macro? main-args make-dir map mat match max member min mod mul multiply name net-accept net-close net-connect net-error net-eval net-listen net-local net-lookup net-peek net-peer net-ping net-receive net-receive-from net-receive-udp net-select net-send net-send-to net-send-udp net-service net-sessions new nil? normal not now nper npv nth nth-set null? number? open or pack parse parse-date peek pipe pmt pop post-url pow pretty-print primitive? print println prob-chi2 prob-z process protected? push put-url pv quote quote? rand random randomize read-buffer read-char read-file read-key read-line real-path ref ref-all ref-set regex remove-dir rename-file replace replace-assoc reset rest reverse rotate round save search seed seek select semaphore sequence series set set-locale set-nth set-ref set-ref-all setq sgn share signal silent sin sinh sleep slice sort source sqrt starts-with string string? sub swap sym symbol? symbols sys-error sys-info tan tanh throw throw-error time time-of-day timer title-case trace trace-highlight transpose trim true? unify unique unless unpack until upper-case uuid wait-pid when while write-buffer write-char write-file write-line xml-error xml-parse xml-type-tags zero? | ~

Testing contexts as objects and scoping rules ...


ALL FUNCTIONS FINISHED SUCCESSFULL: newlisp

total time: 6588

Thanks for the good work!

Peter

PS The "c:\\" with trailing slashes I'll test tomorrow.
_

[newdep]

Heee pjot... you are using the Olga! did you pay for that?
You cant simply steal processor power.. you have to declare those..
Ill fax you the paper...Olga is Ours! You have the Lisa and the Pamela ;-)
Damm competitors ..dies days..

but..nice random catch!

[pjot]

Well our company did take over all CB systems, also the Oga ;-)

I have to send you a bill for using newLisp in Tru64Unix ;-)

[newdep]

I would like to dual Tru64 against our new SUN Blade systems,
we must start a newlisp WarRoom in our server park! ;-)