mingw64 build patch

Machine-specific discussion
Unix, Linux, OS X, OS/2, Windows, ..?
Locked
kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

mingw64 build patch

Post by kosh »

Hi Lutz.

I wrote patch file for builds with MinGW-w64 gcc.
Changes are following:

- Define UINT as pointer size integer (stdint.h, uintptr_t)
- Use type conversion macros for printf (inttypes.h, PRIdPTR)
- Auto detect 64-bit memory model (LP64/LLP64) using C pre-defined macros
- nl-sock.c: bugfix non initialized var DefaultIn

Patch files can be viewed on GitHub:
https://github.com/kosh04/newlisp/compa ... /mingw-w64

(p.s. BBS attachment fails with message The extension diff is not allowed.)

Build and Test (`make testall`) passed under the following platform:

- Win7 Home amd64
-- TDM-GCC (mingw-w64 gcc 4.9.1)
-- MinGW gcc 4.8.1
-- cygwin32 gcc 4.9.2
-- cygwin64 gcc 4.9.2
- Mac OSX clang

However, this patch have some problems:

- stdint.h, inttypes.h headers available in ISO C99. but newLISP C source seem to C90.
- win32-path.c: an incomplete compatibility between `struct stat` and `struct _stat`.
- nl-sock.c: UINT type is name conflict in ws2tcpip.h.
This patch provides the temporary workaround using push_macro/pop_macro.
If you don't like this way, better to rename the UINT to another name (e.g. `typedef uintptr_t nl_int;`)

All reviews are welcome. Thanks.

kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

Re: mingw64 build patch

Post by kosh »


Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Thanks for this great work! Everything worked right away on all OS.

All integrated for 10.6.3 here: http://www.newlisp.org/downloads/develo ... nprogress/
(several reposts after initial post)

Tested on: OS X 10.10, Ubuntu Linux 14.04, FreeBSD 9.3, OpenBSD 5.4 and Windows XP sp2 with mingw32.

I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.

kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

Re: mingw64 build patch

Post by kosh »

Thanks Lutz for merge patch.

I found a missing code in newlisp.h (newlisp-10.6.3.tgz)

Code: Select all

#undef NEWLISP64
#if defined(_LP64) || defined(_WIN64)
#define NEWLISP64
#endif
and qa-specific-tests/qa-local-domain (for `make testall`)

Code: Select all

diff --git a/qa-specific-tests/qa-local-domain b/qa-specific-tests/qa-local-domain
index caa5901..49b41f7 100755
--- a/qa-specific-tests/qa-local-domain
+++ b/qa-specific-tests/qa-local-domain
@@ -6,6 +6,10 @@
 (println)
 (println "Testing UNIX local domain sockets")
 
+(when (find ostype '("Win32" "OS/2"))
+  (println "not tested on " ostype)
+  (exit))
+
 (define (listener path)
 	(set 'lsock (net-listen path))
 	(set 'csock (net-accept lsock))
NEWLISP64 is defined in files makefile_* in this time.
It will more better way to use predefined compiler macros.

Code: Select all

# check predefined macros
$ echo | gcc -m32 -dM -E - | grep WIN64

$ echo | gcc -m64 -dM -E - | grep WIN64
#define _WIN64 1
#define __WIN64 1
#define WIN64 1
#define __WIN64__ 1
Lutz wrote:I wonder what the speed comparison is between the 32-bit and 64-bit versions of newLISP on Windows.
FYI, my AppVeryor's build log info is the following (mingw cross-compile on cygwin):
https://ci.appveyor.com/project/kosh04/ ... ild/1.0.62
Free plans run builds on VM instances with 1 virtual core, 1.75 GB of memory and 127 GB of disk space (Windows Azure "Small").
http://www.appveyor.com/docs/build-configuration

Code: Select all

# 32-bit
>>>>> total time: 14779.53999999999
>>>>> Performance ratio: 5.70 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)

# 64-bit
>>>>> total time: 12015.73600000001
>>>>> Performance ratio: 4.64 (1.0 on MacOSX 10.9, 2.3GHz Intel Core i5, newLISP v10.6.0-64-bit)
My build environment info will be posted at a later.

Regards.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Thanks Kosh. The speed difference between Win32 and Win64 seems to be similar to other platforms.

Currently the selection between 32 and 64 bit is made by the configure script, which then copies one of the makefile_xxx to makefile_build. This gives people the possibility to decide for a different memory model by picking a different makefile_xxxx. This happens often when building for embedded or other resource constrained systems. So, at this point automatic NEWLISP64 detection will not happen yet, but may be added in the future.

The configure-alt script from Ted Walther works a bit different creating makefile_build by building it from scratch and doing more automation in discovering features on different OS platforms. Win64 capability has not yet been added, but that should be easy.

I added detection of Windows 64-bit to configure but currently don’t have the possibility to test it, as my Windows development platforms currently is only 32-bit. I hope the script and new makefiles work for you.

Many qa-specific-test/qa-xxx files have been changed testing for Win64, where appropriate. A few makefiles for mingw64 have been added and should be picked out automatically by the configure script. After fresh install of the source distribution or after a ‘make clean’ a simple ‘make’ will call the configure script first then make the newlisp executable.

Let me know if all this works when using mingw64 tools.

Ps: some more fixes uploaded at 2015-2-18 15:47 UTC

kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

Re: mingw64 build patch

Post by kosh »

makefile_mingwLLP64_utf8 is here:

Code: Select all

# makefile for newLISP v. 10.x.x on MinGW-w64 with UTF-8 support

OBJS = newlisp.o nl-symbol.o nl-math.o nl-list.o nl-liststr.o nl-string.o nl-filesys.o nl-sock.o \
	nl-import.o nl-xml-json.o nl-web.o nl-matrix.o nl-debug.o pcre.o nl-utf8.o win-util.o win-path.o

CFLAGS = -Wall -Wno-uninitialized -Wno-long-long -O1 -g
CPPFLAGS = -DWIN_64 -DSUPPORT_UTF8
#CPPFLAGS = -DWIN_32 -NEWLISP64 -DSUPPORT_UTF8
LDLIBS = -lws2_32

CC = gcc
STRIP = strip

default: newlisp.exe

newlisp.exe: $(OBJS)
	$(CC) $(LDFLAGS) $^ $(LDLIBS) -o $@
	$(STRIP) $@

newlisp.dll: LDFLAGS += -static-libgcc -Wl,--enable-stdcall-fixup
newlisp.dll: $(OBJS) win-dll.o win-dll.def
	$(CC) -shared $(LDFLAGS) $^ $(LDLIBS) -o $@

$(OBJS): primes.h protos.h newlisp.h makefile_mingwLLP64_utf8
This makefile can also make DLL file:

Code: Select all

make -f makefile_mingwLLP64_utf8 newlisp.dll
---
Unfortunatly, new newlisp-10.6.3.tgz compile with errors:

Code: Select all

gcc -Wall -Wno-uninitialized -Wno-long-long -O1 -g -DWIN_64 -DSUPPORT_UTF8  -c -o nl-sock.o nl-sock.c
In file included from nl-sock.c:29:0:

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:160:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN_PKTINFO)==8);

 ^

C:/MinGW64/x86_64-w64-mingw32/include/ws2tcpip.h:167:1: error: size of unnamed array is negative

 C_ASSERT(sizeof(IN6_PKTINFO)==20);

 ^

make: *** [nl-sock.o] Error 1
in nl-sock.c: `#define UINTP uintptr_t` is not ineffective. I recommend to use push_macro/pop_macro, or include newlisp.h after ws2tcpip.h if you don't rename the type name UINT.

---
I hope the value ostype be always "Win32".
This value should not be distinguished by 32/64 bit as well as in other platform. To determine the 64-bit platform can use sys-info.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Thanks Kosh, I will continue to be integrate Windows 64-bit development when I have a development environment similar to yours, with both mingw32 and mingw64 together and the latest stable release.

The logical would be to use a new ostype string “Windows”, although it would break backwards compatibility with code using “Win32”. Using both “Win32” and “Win64” would leave compatibility but be impractical when running old “Win32” scripts on “Win64” installations, although probably those scripts are compatible.

As soon, as I have a new development environment installed, a decision will be made.

kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

Re: mingw64 build patch

Post by kosh »

Lutz, you have completed the setup of new Windows machine?

As an aside, http://tdm-gcc.tdragon.net/ provide "TDM32 Packages". so it works on 32-bit Windows XP.
Of course, a generated newlisp.exe runs on 64-bit Windows.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Not until April, I will have access to a Windows 7 64-bit machine, but thanks for the tip using TDM gcc. That will make the installation a lot easier. I understand that you can cross compile to 64-bint on a 32-bit machine, but I don't have enough space too install, so it has to wait.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

A Windows 64-bit version made with tdm gcc 64 is now working, but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.

The following makefiles have been added: makefile_tdmgcc64 and makefile_tdmgcc64dll and the builds pass make testall except with qa-net6.

Other makefiles including UTF8 and extended FFI using libffi will follow during the next days.

Currently I am still using Win32 and Win64 for the newLISP variable ostype. May be this will change to Windows (64/32 info in in sys-info), as Kosh is suggesting. When the other 64-bit flavors are ready, a .exe will published too for people to test, who can not compile themselves.

kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

Re: mingw64 build patch

Post by kosh »

> but only for IPv4. Perhaps IPv6 is not supported in the tdm version of mingw gcc? Hve not looked into it yet.

Maybe, It because you use variable `defaultIn` without zero-initialization.

https://github.com/kosh04/newlisp/compa ... 1a5cbfL159

initDefaultInAddr() in nl-sock.c

Code: Select all

-defaultIn = allocMemory(defaultInLen);
+defaultIn = callocMemory(defaultInLen);

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Yes, exactly, thanks Kosh! Updated here: http://www.newlisp.org/downloads/develo ... nprogress/

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

... and updated again.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

64-bit 10.6.3 newLISP executables with and without UTF8 suppport can be found here:

http://www.newlisp.org/downloads/development/inprogress

Both .exe are compiled on Windows 7 using mingw64 gcc.

xytroxon
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm
Contact:

Re: mingw64 build patch

Post by xytroxon »

Hi Lutz!

This Win64 version is noticeably faster than the Win32 version - like 7 times faster for some string operations!!!

Then I noticed I was having some missing output using either newlisp64.exe or newlisp64utf8.exe.

Assuming an 80 character x 25 line console display - the following code should display 6000 chars in three sections of 2000 chars each of "+", "-" and "*". But the Win64 newlisp versions will display the first 2000 char "+" section, and of the second 2000 char "-" section, only about the first 50 chars are displayed, and nothing of the third 2000 char "*" section. Sometimes the end of the last line will also show 5 or 6 random chars from the 0 to 31 and 127 to 255 ranges of ascii characters - like a wild pointer running on into another memory variable.

The string length correctly shows all 6000 chars of the "test" string being present, and this code works fine with the last Win32 release of newlisp.

Code was tested on both "console2" and "conemu" Windows console emulator programs with simular results.

Code: Select all

(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(print (length test) "\n" test)
(exit)
-- xytroxon
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976

xytroxon
Posts: 296
Joined: Tue Nov 06, 2007 3:59 pm
Contact:

Re: mingw64 build patch

Post by xytroxon »

Doing a write-file of the "test" string shows no premature shortening of the string or any extra "odd" chars in the file on disk.

Code: Select all

(write-file "test.txt" test)
But reading and printing the file has the same display failure.

Code: Select all

(print (read-file "test.txt"))
So maybe the problem lies within the print function.

--xytroxon
"Many computers can print only capital letters, so we shall not use lowercase letters."
-- Let's Talk Lisp (c) 1976

TedWalther
Posts: 608
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC
Contact:

Re: mingw64 build patch

Post by TedWalther »

I remember a few years ago, Phillip Whelan was able to very quickly locate and patch a 64 bitness problem in newLisp. Wish he was still around, this sounds like a 64 bitness type of problem, right up his alley.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Cannot repeat this problem. Your test code or printing any other string >= 6000 characters always works on Windows 7 Home Premium SP1 command shell cmd.exe or using the MSYS/MinGW bash shell. I wonder if other Windows 7 users can repeat this problem. What version of Windows are you using?

HPW
Posts: 1390
Joined: Thu Sep 26, 2002 9:15 am
Location: Germany
Contact:

Re: mingw64 build patch

Post by HPW »

Hello,

I tried it on my win7 home premium tablet.
When I paste it line by line into the newlisp Shell it works as expected.

But when i make a file Test.lsp with:

Code: Select all

(setq a (string (dup "+" (* 80 25))))
(setq b (string (dup "-" (* 80 25))))
(setq c (string (dup "*" (* 80 25))))
(setq test (string a b c))

(silent(print (length test) "\n" test))
and do a (load "Test.lsp") in a fresh started Shell I get:

Code: Select all

newLISP v.10.6.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (load "Test.lsp")
6000
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------¦.6Å-ï?
This seems to be Buggy.

Edit: The 32 bit Exe does this correctly.

Regards
Hans-Peter
Hans-Peter

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Yes, now I can repeat it too when loading the test code from a file. I could not repeat it when testing the code interactively, but it also fails when interactively when using silent.

It also fails on the 32-bit version in 10.6.3. It seems that this has nothing to too with 32-bit versus 64-bit, but is related to other changes in 10.3.6 development. I am glad we are catching this before 10.6.3 goes out of "inprogress".

kosh
Posts: 72
Joined: Sun Sep 13, 2009 5:38 am
Location: Japan
Contact:

Re: mingw64 build patch

Post by kosh »

I also confirmed this IO problem.

Code: Select all

$ newlisp64.exe -n
> (setf str (string (dup {X} 2049) {HELLO}))
[text]XXXX...<repeat 2049 times>HELLO[/text]
> (println str)
XXXX...<repeat 2049 times>
It seems that cause internal output function `my_vasprintf` (in nl-filesys.c, called from varPrintf)

`vasprintf` has been implemented in mingw64 in stdio.h (but mingw32 yet).
then, It could be fixed in this way:

Code: Select all

/* newlisp.h */
#ifdef WINDOWS
#ifndef __MINGW64__ /* macros depends on mingw implementation */
# define MY_VASPRINTF
# define vasprintf my_vasprintf
#endif
...
#endif
Regards.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: mingw64 build patch

Post by Lutz »

Yes, it was vasprintf, thanks Kosh:

http://www.newlisp.org/downloads/develo ... nprogress/

ValentiMr
Posts: 3
Joined: Tue Aug 27, 2019 12:18 am
Location: Ukraine

-

Post by ValentiMr »

Attached is a patch to fix the test build on head. All it does it make it so the tests compile: they still don't pass at least they don't pass on mac os x.

Also, I've noticed a bunch of build issues on Mac OS X. Anyone mind if I fix those?



Locked