mingw64 build patch

[kosh]

Hi Lutz.

I wrote patch file for builds with MinGW-w64 gcc.
Changes are following:

- Define UINT as pointer size integer (stdint.h, uintptr_t)
- Use type conversion macros for printf (inttypes.h, PRIdPTR)
- Auto detect 64-bit memory model (LP64/LLP64) using C pre-defined macros
- nl-sock.c: bugfix non initialized var DefaultIn

Patch files can be viewed on GitHub:
https://github.com/kosh04/newlisp/compa ... /mingw-w64

(p.s. BBS attachment fails with message The extension diff is not allowed.)

Build and Test (`make testall`) passed under the following platform:

- Win7 Home amd64
-- TDM-GCC (mingw-w64 gcc 4.9.1)
-- MinGW gcc 4.8.1
-- cygwin32 gcc 4.9.2
-- cygwin64 gcc 4.9.2
- Mac OSX clang

However, this patch have some problems:

- stdint.h, inttypes.h headers available in ISO C99. but newLISP C source seem to C90.
- win32-path.c: an incomplete compatibility between `struct stat` and `struct _stat`.
- nl-sock.c: UINT type is name conflict in ws2tcpip.h.
This patch provides the temporary workaround using push_macro/pop_macro.
If you don't like this way, better to rename the UINT to another name (e.g. `typedef uintptr_t nl_int;`)

All reviews are welcome. Thanks.

[kosh]

Patch files are here:
https://gist.github.com/kosh04/6a9ff701d5d6358d3ad9
https://rawgit.com/kosh04/6a9ff701d5d6358d3ad9/raw/

It based newlisp v.10.6.2

[Lutz]

Thanks for this great work! Everything worked right away on all OS.

All integrated for 10.6.3 here: http://www.newlisp.org/downloads/develo ... nprogress/
(several reposts after initial post)

Tested on: OS X 10.10, Ubuntu Linux 14.04, FreeBSD 9.3, OpenBSD 5.4 and Windows XP sp2 with mingw32.

I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.

[kosh]

Thanks Lutz for merge patch.

I found a missing code in newlisp.h (newlisp-10.6.3.tgz)

Code: Select all

#undef NEWLISP64
#if defined(_LP64) || defined(_WIN64)
#define NEWLISP64
#endif


and qa-specific-tests/qa-local-domain (for `make testall`)

Code: Select all

diff --git a/qa-specific-tests/qa-local-domain b/qa-specific-tests/qa-local-domain
index caa5901..49b41f7 100755
--- a/qa-specific-tests/qa-local-domain
+++ b/qa-specific-tests/qa-local-domain
@@ -6,6 +6,10 @@
 (println)
 (println "Testing UNIX local domain sockets")
 
+(when (find ostype '("Win32" "OS/2"))
+  (println "not tested on " ostype)
+  (exit))
+
 (define (listener path)
    (set 'lsock (net-listen path))
    (set 'csock (net-accept lsock))


NEWLISP64 is defined in files makefile_* in this time.
It will more better way to use predefined compiler macros.

Code: Select all

# check predefined macros
$ echo | gcc -m32 -dM -E - | grep WIN64

$ echo | gcc -m64 -dM -E - | grep WIN64
#define _WIN64 1
#define __WIN64 1
#define WIN64 1
#define __WIN64__ 1


I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.

FYI, my AppVeryor's build log info is the following (mingw cross-compile on cygwin):
https://ci.appveyor.com/project/kosh04/ ... ild/1.0.62

Free plans run builds on VM instances with 1 virtual core, 1.75 GB of memory and 127 GB of disk space (Windows Azure "Small").
http://www.appveyor.com/docs/build-configuration

Code: Select all

# 32-bit
>>>>> total time: 14779.53999999999
>>>>> Performance ratio: 5.70 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)

# 64-bit
>>>>> total time: 12015.73600000001
>>>>> Performance ratio: 4.64 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)


My build environment info will be posted at a later.

Regards.

[Lutz]

Thanks Kosh. The speed difference between Win32 and Win64 seems to be similar to other platforms.

Currently the selection between 32 and 64 bit is made by the configure script, which then copies one of the makefile_xxx to makefile_build. This gives people the possibility to decide for a different memory model by picking a different makefile_xxxx. This happens often when building for embedded or other resource constrained systems. So, at this point automatic NEWLISP64 detection will not happen yet, but may be added in the future.

The configure-alt script from Ted Walther works a bit different creating makefile_build by building it from scratch and doing more automation in discovering features on different OS platforms. Win64 capability has not yet been added, but that should be easy.

I added detection of Windows 64-bit to configure but currently don’t have the possibility to test it, as my Windows development platforms currently is only 32-bit. I hope the script and new makefiles work for you.

Many qa-specific-test/qa-xxx files have been changed testing for Win64, where appropriate. A few makefiles for mingw64 have been added and should be picked out automatically by the configure script. After fresh install of the source distribution or after a ‘make clean’ a simple ‘make’ will call the configure script first then make the newlisp executable.

Let me know if all this works when using mingw64 tools.

Ps: some more fixes uploaded at 2015-2-18 15:47 UTC

[kosh]

makefile_mingwLLP64_utf8 is here:

Code: Select all

# makefile for newLISP v. 10.x.x on MinGW-w64 with UTF-8 support

OBJS = newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o \
   nl-import.o nl-xml-json.o nl-web.o nl-matrix.o nl-debug.o pcre.o nl-utf8.o win-util.o win-path.o

CFLAGS = -Wall -Wno-uninitialized -Wno-long-long -O1 -g
CPPFLAGS = -DWIN_64 -DSUPPORT_UTF8
#CPPFLAGS = -DWIN_32 -NEWLISP64 -DSUPPORT_UTF8
LDLIBS = -lws2_32

CC = gcc
STRIP = strip

default: newlisp.exe

newlisp.exe: $(OBJS)
   $(CC) $(LDFLAGS) $^ $(LDLIBS) -o $@
   $(STRIP) $@

newlisp.dll: LDFLAGS += -static-libgcc -Wl,--enable-stdcall-fixup
newlisp.dll: $(OBJS) win-dll.o win-dll.def
   $(CC) -shared $(LDFLAGS) $^ $(LDLIBS) -o $@

$(OBJS): primes.h protos.h newlisp.h makefile_mingwLLP64_utf8


This makefile can also make DLL file:

Code: Select all

make -f makefile_mingwLLP64_utf8 newlisp.dll

---
Unfortunatly, new newlisp-10.6.3.tgz compile with errors:

Code: Select all

gcc -Wall -Wno-uninitialized -Wno-long-long -O1 -g -DWIN_64 -DSUPPORT_UTF8  -c -o nl-sock.o nl-sock.c
In file included from nl-sock.c:29:0:

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:160:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN_PKTINFO)==8);

 ^

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:167:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN6_PKTINFO)==20);

 ^

make: *** [nl-sock.o] Error 1


in nl-sock.c: `#define UINTP uintptr_t` is not ineffective. I recommend to use push_macro/pop_macro, or include newlisp.h after ws2tcpip.h if you don't rename the type name UINT.

---
I hope the value ostype be always "Win32".
This value should not be distinguished by 32/64 bit as well as in other platform. To determine the 64-bit platform can use sys-info.

[Lutz]

Thanks Kosh, I will continue to be integrate Windows 64-bit development when I have a development environment similar to yours, with both mingw32 and mingw64 together and the latest stable release.

The logical would be to use a new ostype string “Windows”, although it would break backwards compatibility with code using “Win32”. Using both “Win32” and “Win64” would leave compatibility but be impractical when running old “Win32” scripts on “Win64” installations, although probably those scripts are compatible.

As soon, as I have a new development environment installed, a decision will be made.

[kosh]

Lutz, you have completed the setup of new Windows machine?

As an aside, http://tdm-gcc.tdragon.net/ provide "TDM32 Packages". so it works on 32-bit Windows XP.
Of course, a generated newlisp.exe runs on 64-bit Windows.

[Lutz]

Not until April, I will have access to a Windows 7 64-bit machine, but thanks for the tip using TDM gcc. That will make the installation a lot easier. I understand that you can cross compile to 64-bint on a 32-bit machine, but I don't have enough space too install, so it has to wait.

[Lutz]

A Windows 64-bit version made with tdm gcc 64 is now working, but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.

The following makefiles have been added: makefile_tdmgcc64 and makefile_tdmgcc64dll and the builds pass make testall except with qa-net6.

Other makefiles including UTF8 and extended FFI using libffi will follow during the next days.

Currently I am still using Win32 and Win64 for the newLISP variable ostype. May be this will change to Windows (64/32 info in in sys-info), as Kosh is suggesting. When the other 64-bit flavors are ready, a .exe will published too for people to test, who can not compile themselves.

[kosh]

> but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.

Maybe, It because you use variable `defaultIn` without zero-initialization.

https://github.com/kosh04/newlisp/compa ... 1a5cbfL159

initDefaultInAddr() in nl-sock.c

Code: Select all

-defaultIn = allocMemory(defaultInLen);
+defaultIn = callocMemory(defaultInLen);

[Lutz]

Yes, exactly, thanks Kosh! Updated here: http://www.newlisp.org/downloads/develo ... nprogress/

[Lutz]

... and updated again.

[Lutz]

64-bit 10.6.3 newLISP executables with and without UTF8 suppport can be found here:

http://www.newlisp.org/downloads/development/inprogress

Both .exe are compiled on Windows 7 using mingw64 gcc.

[xytroxon]

Hi Lutz!

This Win64 version is noticeably faster than the Win32 version - like 7 times faster for some string operations!!!

Then I noticed I was having some missing output using either newlisp64.exe or newlisp64utf8.exe.

Assuming an 80 character x 25 line console display - the following code should display 6000 chars in three sections of 2000 chars each of "+", "-" and "*". But the Win64 newlisp versions will display the first 2000 char "+" section, and of the second 2000 char "-" section, only about the first 50 chars are displayed, and nothing of the third 2000 char "*" section. Sometimes the end of the last line will also show 5 or 6 random chars from the 0 to 31 and 127 to 255 ranges of ascii characters - like a wild pointer running on into another memory variable.

The string length correctly shows all 6000 chars of the "test" string being present, and this code works fine with the last Win32 release of newlisp.

Code was tested on both "console2" and "conemu" Windows console emulator programs with simular results.

Code: Select all

(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(print (length test) "\n" test)
(exit)


-- xytroxon

[xytroxon]

Doing a write-file of the "test" string shows no premature shortening of the string or any extra "odd" chars in the file on disk.

Code: Select all

(write-file "test.txt" test)


But reading and printing the file has the same display failure.

Code: Select all

(print (read-file "test.txt"))


So maybe the problem lies within the print function.

--xytroxon

[TedWalther]

I remember a few years ago, Phillip Whelan was able to very quickly locate and patch a 64 bitness problem in newLisp. Wish he was still around, this sounds like a 64 bitness type of problem, right up his alley.

[Lutz]

Cannot repeat this problem. Your test code or printing any other string >= 6000 characters always works on Windows 7 Home Premium SP1 command shell cmd.exe or using the MSYS/MinGW bash shell. I wonder if other Windows 7 users can repeat this problem. What version of Windows are you using?

[HPW]

Hello,

I tried it on my win7 home premium tablet.
When I paste it line by line into the newlisp Shell it works as expected.

But when i make a file Test.lsp with:

Code: Select all

(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(silent(print (length test) "\n" test))


and do a (load "Test.lsp") in a fresh started Shell I get:

Code: Select all

newLISP v.10.6.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (load "Test.lsp")
6000
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------¦.6Å-ï?


This seems to be Buggy.

Edit: The 32 bit Exe does this correctly.

Regards
Hans-Peter

[Lutz]

Yes, now I can repeat it too when loading the test code from a file. I could not repeat it when testing the code interactively, but it also fails when interactively when using silent.

It also fails on the 32-bit version in 10.6.3. It seems that this has nothing to too with 32-bit versus 64-bit, but is related to other changes in 10.3.6 development. I am glad we are catching this before 10.6.3 goes out of "inprogress".

[kosh]

I also confirmed this IO problem.

Code: Select all

$ newlisp64.exe -n
> (setf str (string (dup {X} 2049) {HELLO}))
[text]XXXX...<repeat 2049 times>HELLO[/text]
> (println str)
XXXX...<repeat 2049 times>


It seems that cause internal output function `my_vasprintf` (in nl-filesys.c, called from varPrintf)

`vasprintf` has been implemented in mingw64 in stdio.h (but mingw32 yet).
then, It could be fixed in this way:

Code: Select all

/* newlisp.h */
#ifdef WINDOWS
#ifndef __MINGW64__ /* macros depends on mingw implementation */
# define MY_VASPRINTF
# define vasprintf my_vasprintf
#endif
...
#endif


Regards.

[Lutz]

Yes, it was vasprintf, thanks Kosh:

http://www.newlisp.org/downloads/develo ... nprogress/