pay someone for OpenBSD fixes?

Lutz · Post by **Lutz** » Mon Apr 27, 2009 11:18 am

The odbc and sqlite3 modules may need to be updated too, if 64-bit libraries exist for these at all. Other modules in the distribution are probably safe to use with 64-bit newLISP unchanged.

The new mysql.lsp module will be posted later, and then it goes into the 10.0.5 maintenance release in May.

TedWalther · Post by **TedWalther** » Mon Apr 27, 2009 11:23 pm

Yes, there are 64bit libraries for all the standard modules that come with newLISP. The account is still active if you want to do any updates or testing.

crypto.lsp
* /usr/lib/libcrypto.so.14.0

gmp.lsp
* /usr/local/lib/libgmp.so.7.1

mysql.lsp
* /usr/local/lib/libmysqlclient.so.19.0

odbc.lsp
* /usr/local/lib/libiodbc.so.3.15
* /usr/local/lib/libiodbcinst.so.3.15

sqlite3.lsp
* /usr/local/lib/libsqlite3.so.11.0

unix.lsp
* /usr/lib/libc.so.48.0

zlisp.lsp
* /usr/lib/libz.so.4.1

Lutz · Post by **Lutz** » Tue Apr 28, 2009 1:18 pm

I couldn't find any of the following libraries on your system:

* /usr/local/lib/libgmp.so.7.1
* /usr/local/lib/libiodbc.so.3.15
* /usr/local/lib/libiodbcinst.so.3.15
* /usr/local/lib/libsqlite3.so.11.0

perhaps you looked on a 32-bit machine, not on the 64-bit? But I could verify the correct working of libc, libcrypto and libz under 64-bit newLISP.

There is a new 10.0.5 on your system and I accidently removed 10.0.4-local. I hope your local adaptations can quickly be redone.

Changes are made for all the special OpenBSD compiler warnings, but one warning still shows up although already fixed. Perhaps some cacheing issue? Rebooting the machine will probably make it go away.

TedWalther · Post by **TedWalther** » Tue Apr 28, 2009 5:08 pm

Thanks, I was looking at those libraries on another 64bit OpenBSD machine. They are now installed on the test box.

TedWalther · Post by **TedWalther** » Tue Apr 28, 2009 6:34 pm

The warnign was referring to your use of strncat(); openbsd deprecates it as much as strcat() in favor of strlcat (and strlcpy). I'm now looking into how to incorporate those two functions portably.

Lutz · Post by **Lutz** » Tue Apr 28, 2009 7:16 pm

Thanks Ted, all warnings are gone now and libgmp and libsqlite3 work too. Only sqlite3.lsp needed to adapt to 64-bit. The gmp.lsp module worked right away with the added library path for OpenBSD.

Adapting the odbc.lsp to 64-bit will take longer and may not happen for the next release. It never has been used on UNIX anyway and lacks functions for creating ODBC instances. On Win32, for which this module was created originally, ODBC instances are created using a control applet.

Thanks again for letting me use your OpenBSD system. If you could leave the account open for me another few weeks, that would be great. I could then more thoroughly test the upcoming maintenance release.

ps: note that mysql.lsp still contains the logon credentials on your machine.

TedWalther · Post by **TedWalther** » Tue Apr 28, 2009 9:58 pm

I'm happy to make the box available as long as you'd like for continuing to make newlisp the best programming environment ever. As long as the login credentials don't get into the next release, that is alright ;-) I just made a patch to get rid of two warnings:

First, the strlcat() warning is now fixed, I updated the source to use those functions, with appropriate error checking.

Next, in the command generator for the readline support, I added some error checking and the (char*) cast that shut the compiler up.

Does the patch suit your style sufficiently?

http://reactor-core.org/newlisp.diff.txt

Lutz · Post by **Lutz** » Tue Apr 28, 2009 11:10 pm

I am more selective on error-checking and would prefer to not have it in those places, where you added it ;-). Error-checking policy, as it is now, has proven to be more than sufficient in the past. I also prefer to stay with the strlcat() and strlcpy() versions built into gcc. The old code using strcpy() and strcat() was already safe, and the move to strlcpy() and strlcat() was made only to suppress compiler warnings.

It would be great to have that OpenBSD machine available to me in the future. If you have to change credentials once in a while, that is Ok with me, just let me know when you make a change. I have similar arrangements with other users (e.g. Solaris and TRU64) and it has helped to make newLISP safer and stable. OpenBSD is specially interesting, because they put this focus on security and a clean compile is harder to achieve than on other OSs.

Thanks again for making OpenBSD 64-bit available to me.

TedWalther · Post by **TedWalther** » Wed Apr 29, 2009 12:00 am

Lutz wrote:I am more selective on error-checking and would prefer to not have it in those places, where you added it ;-). Error-checking policy, as it is now, has proven to be more than sufficient in the past. I also prefer to stay with the strlcat() and strlcpy() versions built into gcc. The old code using strcpy() and strcat() was already safe, and the move to strlcpy() and strlcat() was made only to suppress compiler warnings.

It would be great to have that OpenBSD machine available to me in the future. If you have to change credentials once in a while, that is Ok with me, just let me know when you make a change. I have similar arrangements with other users (e.g. Solaris and TRU64) and it has helped to make newLISP safer and stable. OpenBSD is specially interesting, because they put this focus on security and a clean compile is harder to achieve than on other OSs.

Thanks again for making OpenBSD 64-bit available to me.

The box is available to you as long as you need. It is also multicore running SMP, in case that is needed.

strlcat and strlcpy aren't included in GNU libc, the OpenBSD guys recommended direct inclusion of the functions as the portable way to do it.

Lutz · Post by **Lutz** » Wed Apr 29, 2009 10:50 am

yes, I just realize strlcat() and strlcpy() are only on BSD based OSs (Mac OSX, FreeBSD, Solaris) but not on Linux and Win32. In those cases strncpy() and strncat() can stay as the code was safe already (even using strcpy() and strcat()), and no warnings are generated on these platforms.

The box is available to you as long as you need. It is also multicore running SMP, in case that is needed.

Thanks, and SMP is good news for launching parallel processes using:

http://www.newlisp.org/newlisp_manual.html#spawn

Mac OSX does a good job balancing multiple processes launched by 'spawn' on multi-core CPUs and I guess OpenBSD will do similar with SMP.

Lutz · Post by **Lutz** » Wed Apr 29, 2009 11:31 am

... turns out BSD SMP does an even better job balancing three processes on 2 AMD64 processors or cores (I assume) compared to OSX on Mac Mini 1.83 GHZ Intel Core Duo:

7478ms vs 5142ms on BSD -> 1.45 (BSD 64-bit, newLISP 64-bit)

17101ms vs 12574ms on OSX -> 1.36 (OSX 64-bit, newLISP 32-bit)

13535ms vs 9925ms on OSX -> 1.36 (OSX 64-bit, newLISP 64-bit)

measured with the 'spawn' prime number example in the manual.

TedWalther · Post by **TedWalther** » Wed Apr 29, 2009 4:41 pm

Lutz wrote:... turns out BSD SMP does an even better job balancing three processes on 2 AMD64 processors or cores (I assume) compared to OSX on Mac Mini 1.83 GHZ Intel Core Duo:

7478ms vs 5142ms on BSD -> 1.45 (BSD 64-bit, newLISP 64-bit)

17101ms vs 12574ms on OSX -> 1.36 (OSX 64-bit, newLISP 32-bit)

13535ms vs 9925ms on OSX -> 1.36 (OSX 64-bit, newLISP 64-bit)

measured with the 'spawn' prime number example in the manual.

Actually it is a quad-core box; that might make a difference.

itistoday · Post by **itistoday** » Thu Apr 30, 2009 3:53 am

From what I understand there are lots of improvements to be made in the OS X kernel in relation to SMP, and that many of these improvements should find their way into Snow Leopard. Great discussion btw! Thanks Lutz for getting that mysql module to work on 64-bit!

Lutz · Post by **Lutz** » Fri May 01, 2009 12:13 am

As always, benchmarks are a tricky business with all sorts of surprises.

After Ted revealed that his machine is actually a quad core (not dual as I assumed), I tried a spawn with four processes, increasing the ratio from 1.45 to 1.58 (simple versus spawn) on BSD but on OS X dual core I now get even better to a 1.67 ratio!