It would be nice to have a 'make test' or 'make check' option
after running 'make' when building Newlisp. It could help speed
debugging on "non-supported" platforms like Solaris.
I'm willing to be an "official" Solaris tester if this would help.
I'm thinking of something along the lines of the test suite
in php and a number of other projects.
How about a test suite?
-
nigelbrown
- Posts: 429
- Joined: Tue Nov 11, 2003 2:11 am
- Location: Brisbane, Australia
For interest see a few earlier comments on testing
http://www.alh.net/newlisp/phpbb/viewtopic.php?t=99
Nigel
http://www.alh.net/newlisp/phpbb/viewtopic.php?t=99
Nigel
Such a test suite already exists and is shipped with the source distribution. It does check most functions. Many I/O functions are not nested, I test those with applications, but a testsuite should be written for it too.
The file is called 'qa' (for Quality Assurance) and is in the main directory. After each build I do a:
./newlisp qa
The file automatically goes through every built in function and calls it, ie. the function 'print' is called as (test-print). So the file contains definitions for all functions to be tested. Some functions don't have a full routine yet and are defined as:
(define (test-xxx) true) ;; where xxx is the function name
Every test functions must return 'true' or it will not pass and the qa routine will stop. If the whole suite finishes with the word "Finished" it has passed.
At the end of the suite I have a special part testing all kinds of weird situations for contexts. I have other shorter scripts, not included in the distribution to stress test the symbol implementation (which has nt changed for years).
Another condition of the qa file is that it must be possible to do:
(dotimes (x N) (load "qa"))
For stress testing and finding memory leaks. I run this not on every release, but is has been done on 8.1.0. This test will pass clean on Linux with thousands of iterations, but not on Win32, which has a leak opening / closing file resources. Looping over QA means that certain things have to be cleaned up or initialized after/before runs. The qa suite creates several files.
Every official release is tested on:
Linux Mandrake,
Linux Debian,
Linux Readhat,
Linux AMD64
Solaris 9 x86 (very few times Sparc)
FreeBSD (few times also OpenBSD and NetBSD)
Mac OS X 10.2
Win32 MinGW
Win32 Cygwin
Win32 Borland
Note that some functions are outcommented on some OS, like 'import', or 'pipe'.
Development releases are typically only tested on:
Linux Mandrake
Free BSD
MinGW
UTF-8 is tested on
MinGW
Linux Mandrake
The Linux Library on:
newlisp.dll with MinGW
newlisp.so with Linux
It would be great if people could contribute more of the test-xxx functions and I can easily integrate them in QA.
Testing functionality should contain the following in priority order:
(1) Test normal functionality as described in the manual
(2) Test border conditions of parameters (including missing parameters) like small numbers 0 big numbers empty strings etc.
(3) Test 'weird' input conditions, try to 'break' it, i.e. negative or zero arguments although not allowed, wrong data types, etc.
(4) Stress testing with many iterations, loaded memory conditions etc.
On stress testing, you start testing mostly the OS itself reacting on certain newLISP demands, i.e. Linux never crashes on memory stress tests, but Win32 does.
Over time I have added test-cases for reported bugs. I.e. after search was reported to crash on a cosed filehandle, I *should* have added this as a testcase. Right now I/O functions are not tested.
One should also test on mayor own developed applications, i.e. I test every release on the newlisp.orh website, which is completely scripted in newLISP including spamfilters cron jobs log-reporting etc.
As you see its a big and important field and every help is appreciated.
Lutz
The file is called 'qa' (for Quality Assurance) and is in the main directory. After each build I do a:
./newlisp qa
The file automatically goes through every built in function and calls it, ie. the function 'print' is called as (test-print). So the file contains definitions for all functions to be tested. Some functions don't have a full routine yet and are defined as:
(define (test-xxx) true) ;; where xxx is the function name
Every test functions must return 'true' or it will not pass and the qa routine will stop. If the whole suite finishes with the word "Finished" it has passed.
At the end of the suite I have a special part testing all kinds of weird situations for contexts. I have other shorter scripts, not included in the distribution to stress test the symbol implementation (which has nt changed for years).
Another condition of the qa file is that it must be possible to do:
(dotimes (x N) (load "qa"))
For stress testing and finding memory leaks. I run this not on every release, but is has been done on 8.1.0. This test will pass clean on Linux with thousands of iterations, but not on Win32, which has a leak opening / closing file resources. Looping over QA means that certain things have to be cleaned up or initialized after/before runs. The qa suite creates several files.
Every official release is tested on:
Linux Mandrake,
Linux Debian,
Linux Readhat,
Linux AMD64
Solaris 9 x86 (very few times Sparc)
FreeBSD (few times also OpenBSD and NetBSD)
Mac OS X 10.2
Win32 MinGW
Win32 Cygwin
Win32 Borland
Note that some functions are outcommented on some OS, like 'import', or 'pipe'.
Development releases are typically only tested on:
Linux Mandrake
Free BSD
MinGW
UTF-8 is tested on
MinGW
Linux Mandrake
The Linux Library on:
newlisp.dll with MinGW
newlisp.so with Linux
It would be great if people could contribute more of the test-xxx functions and I can easily integrate them in QA.
Testing functionality should contain the following in priority order:
(1) Test normal functionality as described in the manual
(2) Test border conditions of parameters (including missing parameters) like small numbers 0 big numbers empty strings etc.
(3) Test 'weird' input conditions, try to 'break' it, i.e. negative or zero arguments although not allowed, wrong data types, etc.
(4) Stress testing with many iterations, loaded memory conditions etc.
On stress testing, you start testing mostly the OS itself reacting on certain newLISP demands, i.e. Linux never crashes on memory stress tests, but Win32 does.
Over time I have added test-cases for reported bugs. I.e. after search was reported to crash on a cosed filehandle, I *should* have added this as a testcase. Right now I/O functions are not tested.
One should also test on mayor own developed applications, i.e. I test every release on the newlisp.orh website, which is completely scripted in newLISP including spamfilters cron jobs log-reporting etc.
As you see its a big and important field and every help is appreciated.
Lutz
Last edited by Lutz on Wed Aug 18, 2004 1:37 pm, edited 4 times in total.
Note to Gregben (is it Greg?) about Solaris testing:
Yes, if you could do more tests on Solaris that would be great, are you running on a Sparc right? I test at Sourceforge on Solaris on an x86. I can compile and link on their Sparc without warnings, but at last I get the following warning:
BFD: stPDaWqF: warning: allocated section `.interp' not in segment
when I try running it I get:
newlisp: Cannot find ⌂ELF
Killed
I know that others have built fine on Sparc, so I guess it is a problem with the installation at Sourceforge. 'pipe' seems not to work correctly on Solaris, but I have not investigated further. May be just an adjustment of the 'sleep' functions to higher values. The Sparc on Sourceforge is very slow (probably overloaded).
I have never tested the 'import' function on Solaris and it is not coded in the QA file, because lib directories everywhere are to different.
Lutz
Yes, if you could do more tests on Solaris that would be great, are you running on a Sparc right? I test at Sourceforge on Solaris on an x86. I can compile and link on their Sparc without warnings, but at last I get the following warning:
BFD: stPDaWqF: warning: allocated section `.interp' not in segment
when I try running it I get:
newlisp: Cannot find ⌂ELF
Killed
I know that others have built fine on Sparc, so I guess it is a problem with the installation at Sourceforge. 'pipe' seems not to work correctly on Solaris, but I have not investigated further. May be just an adjustment of the 'sleep' functions to higher values. The Sparc on Sourceforge is very slow (probably overloaded).
I have never tested the 'import' function on Solaris and it is not coded in the QA file, because lib directories everywhere are to different.
Lutz