Compile DLL

Q&A's, tips, howto's
Locked
sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Compile DLL

Post by sunmountain »

Hi there,
I got here in test.cpp this

Code: Select all

extern "C"
{
	__declspec(dllexport) double adder(double a,double b) {
		return (a+b);
	}
}
I then compile this with

Code: Select all

C:\Users\stefan\ffi>cl /LD /DLL test.cpp
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

test.cpp
Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test.dll
/dll
/implib:test.lib
test.obj
   Creating library test.lib and object test.exp
test.dll now exports a function adder.
But, when I start newlisp, import the func and start it, i get this:

Code: Select all

newLISP v.10.3.5 on Win32 IPv4/6 UTF-8, execute 'newlisp -h' for more info.

> (import "test.dll" "adder")
adder<10001000>
> (adder 2.2 3.0)
-1717986918
It doesn't matter if I append "cdecl" or not.

I wrote a test prog to see if the dll is ok:

Code: Select all

#include <stdio.h>

double adder(double a,double b);

int main(void) {
	printf("%f\n",adder(3.2,4.5));
	return 0;
}
compiled it

Code: Select all

C:\Users\stefan\ffi>cl testtest.c test.lib
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

testtest.c
Microsoft (R) Incremental Linker Version 9.00.30729.01
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:testtest.exe
testtest.obj
test.lib
and run it:

C:\Users\stefan\ffi>testtest.exe
7.700000

Where is my error ?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

The newLISP import function will pass double floats correctly into cdecl compiled function, but will not return values as doubles. You can either return arrays of double floats or pointers to double floats and then use either 'get-float' or 'unpack' on those addresses. See also here:

http://www.newlisp.org/downloads/CodePa ... tml#toc-23

and here:

http://www.newlisp.org/downloads/newlis ... #get-float

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

I should have read the fine manual before, though.
I guess it has internally to do with the fact, that floats can't be represented as normal numbers
(duality pointer <-> long).

Time for a FFI library :-)

I took a look at several ffi libs, and I think it would be the easiest to extend newLisp's vocabulary for this.
As far as I have seen, it would be neccessary to create something like nl-ffi.c and correspondig
entries in primes.h.

Correct ?

Do you have a process to get (if it becomes usable) this thing into newLisp source ?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

Yes, that is the way to implement new functions in newLISP, and put a prototype of the p_xxxx function into protos.h.

But I think libffi is too big for what newLISP tries to accomplish. You will lose newLISP's small size and short startup time, necessary for parallel and distributed tasks with multiple newLISP instances. That is why I recommend writing stub libraries for the case where the normal 'import' is not enough. Most users will never import a single library when using newLISP. Those who do, will most of the time do fine , with what the 'import' function can deliver.

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

But I think libffi is too big for what newLISP tries to accomplish.
  • Not really. When stripped, it ads about 25 kB.
    I try to accomplish to do the same things with newLisp as with Python: penetration testing, automation, regression tests, gui development etc etc
You will lose newLISP's small size and short startup time, necessary for parallel and distributed tasks with multiple newLISP instances.
  • loosing small size: not really.
    loosing small startup times: I don't think so, for the same reason + the fact, that ffi does not need big initialization procedures
    if I'm doing that sort of stuff, I should be aware of that - and perhaps compile an extended version for newlisp perhaps (if it really needs some sort of alien library), which is really easy IMHO
That is why I recommend writing stub libraries for the case where the normal 'import' is not enough.
  • That is another option. But that would end in "import" again - which means I cannot return or get real newlisp cells etc, which would be benefical for defining structs etc.
    Approx. newlisps vocabulary would grow by about 10 pieces
    it would perfectly possible to have #defines for compiling support for ffi on/off
Most users will never import a single library when using newLISP. Those who do, will most of the time do fine , with what the 'import' function can deliver.
  • I would not kick it out - leave it for simple things, but have ffi for complex(er) things
    Its low hanging fruit :-)
Perhaps a runtime plugin system could be a long term goal.


But, let me see how far I can get this weekend.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

Perhaps you can create a nl-ffi.c and, compile with: gcc -DFFI and have #ifdef FFI for primes.h and protos.h. Then it is easy to turn it on and off. newLISP only supports LP64 and ILP32 memory models, perhaps that helps to cut down on size for FFI stuff. At the moment there is no support for LP64 on Windows, mainly because MinGW doesn't offer it (yet).

I am not completely convinced yet, but I agree that it's worth to give it a shot.

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

Hi Lutz,
Perhaps you can create a nl-ffi.c and, compile with: gcc -DFFI and have #ifdef FFI for primes.h and protos.h. Then it is easy to turn it on and off.
That was (is) exactly my plan.
I'll take 10.3.6 as the base for this.

Due to the fact that I have a Win 7 dev box (and 32 bit only), I'm not able to test LP64 here.
I think it is not only the compiler (there is a MinW-W64), but even so the fact
that the MS runtime ecosystem is LLP64.

And, I as far as I'v seen, newLisp is relatively hard bound to the standard unix os interface,
so porting it to the Microsoft C++ compiler (and the Windows programming model) would be a longer journey.
If at all, it is worth the effort - maintaining a piece of C/C++ for several platforms is one thing (lots of #ifdefs) - for several platforms + several compilers squares the number of #ifdefs.
And it would not solve the LP64/ILP32 issues - it would be even more neccessary to
make the code aware of several memory models (I remember sources, where very ugly things got done
to accomplish such).

And then the beauty of the newLisp source code will be gone forever.

Now it's small, well designed, structured and easy to adopt (it took me 10 min to get my own verbs working).

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

Good to know that MinGW-w64 v1.0 is now available, thanks for the news.

Taking care of LLP64 in Windows 7, will take a little bit more time and testing above all. Most of the changes will be kept local to newlisp.h. Because newLISP is often compiled on small platforms - now even more because of the rise of mobile platforms - both 32-bit and 64-bit versions will need to be supported for a long time.

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

Taking care of LLP64 in Windows 7, will take a little bit more time and testing above all.
That's funny.
I was just wondering today, if there are automated tests as a regression after (automatic) newLisp builds.

Anyway, for the nl-ffi thing I'll write some.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

There is a shorter and a longer test suite. And both can run with more or less reporting. After making the newlisp executable, run from the main distribution directory:

For a short suite and less reporting:
make test

For a short suite with full reporting:
make check

For a long suite with less reporting:
make testall

For a long suite with full reporting:
make checkall

On a modern machine, just do a 'make testall'. I normally do:

make clean; make; make testall

all in a row, separated by semicolons. On Windows, I work in a MSYS Bash shell.

When you write a qa-ffi, write your diagnostics, whatever else, to the console. At the end do a:

(println ">>>>> <your description here> SUCCESSFUL")

or a:

(println ">>>>> PROBLEM <your description here>")

'make test' and 'make testall' basically run check and checkall but only report lines with at least 3 angle brackets ">>>".

You can run a single qa- file, as an example:

./newlisp qa-specific-tests/qa-net6

Most qa-xxx files are in the qa-specific-tests/ directory, but there are two 'qa-xxxx' files in the main directory: 'qa-dot' and 'qa-comma'. Both do a quick test of all functions in the system. qa-dot does it for locales with a decimal "." dot and qa-comma does it for locales with a decimal "," comma.

I also use qa-dot and qa-comma to check for cell or memory leakage. Comment out the last 'exit' statement in the script and report (sys-info). From the newlisp command line, do several (load "qa-dot") to check for cell leakage. After no more than two invocations the cell count should remain constant. Once in a while I use 'valgrind' to check for general memory leaks.

For main releases, I also check some files from the example/ directory and run the old modified Debian benchmark suite and applications offered at newlisp.org (the wiki and the ide).

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

I sat down an made a layout of my code.
To be honest: I think it will be neccessary to create a new cell type (most likely types),
for ffi calls, callbacks and some types.

I'd like to be able to do something like this (later):

Code: Select all

(import "msvrcrt.dll" "printf" "cdecl")
(set 'cprintf (fcall printf c_int (*c_char c_int)))
(cprintf "The answer to all is: %d\n" 42)


or so

Code: Select all

(define (cmp a b) (set 'a (get-int a))(set 'b (get-int b))(if (> a b) 1 (> b a) -1 (= a b) 0))
(set 'qsort (fcall qsort c_int c_int (*c_int c_func)))
(set 'sortcb (fclosure  cmp c_int *c_int *c_int))
(set 'clist (farray c_int (34 2 3 5 6)))
(qsort clist (fsizeof clist) sortcb))
When newlisp tries to execute the line (cprintf ...) it must be able
to figure out how, like with executeLibfunction, which is bound to CELL_IMPORT_[DLL|CDECL].

Some more intrinsic than I thought first.

So, many #ifdefs :-)

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

One additional type CELL_IMPORT_FFI should be enough.

Then 'fcall' registers the call pattern internally. I wonder if it would be possible to use existing symbols for call patterns:

*, string - generic pointer 32-bit or 64-bit depending on the newLISP version
char - one-byte unsigned byte
int - 32-bit
long - 64-bit (new symbol)
float - IEEE 769 double float
flt - 32-bit float

* and string would be used mostly for the same thing, but having both improves readability.

Everything else gets resolved with existing functions like pack, unpack, get-char, get-float, get-int and get-long for breaking up data structures and reformatting return values.

Don't try to cover everything under the sky. We can handle 90% of functions already, if we get to 99% that is enough.

If the thing isn't small enough and easy enough to use, then it's not worth it. Remember a module writer, who has to be knowledgeable in C anyway, would always be able to write a stub to be used with the existing 'import'.

What I mean is, not having FFI doesn't prevent people from writing modules. Having FFI only makes sense if it really is a lot easier to use than writing your import stubs library.

Not trying to discourage you, but I think it's a really hard problem to solve ;-)

Perhaps, starting out the whole thing, writing a stub-wrapper for FFI to be 'inport'ed by newLISP via a module written in newLISP, is a better start:

newlisp-program -> newlisp-module -> C-ffi-wrapper -> libffi.dll/so/dylib -> 3rd-party-lib

Then, after that works, trying to get rid of C-ffi-wrapper. Another possibility would be to modify libffi.dll/so/dylib to our own:

newlisp-program -> newlisp-module -> modified-for-newlisp-libffi.dll/so/dylib -> 3rd-party-lib

finally putting a stripped down modified-for-newlisp-libffi.dll/so/dylib into newLISP:

newlisp-program -> newlisp-module -> 3rd-party-lib

I guess the last one is, what you are shooting for.

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

newlisp-program -> newlisp-module -> 3rd-party-lib

I guess the last one is, what you are shooting for.
Exactly.

About my motivation:
My goal ist to make it easier working with more complex structures.
Mostly these cause more work, due to pack/unpack.

At first I thought it could be done solely with import/pack/unpack and some
define-macro magic in combination with FOOP (create context objects with default functors behaving the right way).
But then I discoverd the restricted number of callbacks, and as I do GUI development quite often,
that would be a real constrain (which could even be solved otherwise, sure.).
I thought it would be cool to have a generic way of creating something the ffi library calls
closures, so having one trampoline function which does name based dispatching and transforming
the parameters and return values transparently to the newLisp programmer.
And: I want to learn :-)
And yes, it's hard work at a first glance - but the newLisp code makes it easy - at least in theory.

Anyway, could you please explain if there is a standard pattern on using the CELL members
aux and contents ?

If I'm creating CELL_FFI_IMPORT type, I could for example do a dispatching over aux
and store pointers to concrete objects in contents, so I would only need one CELL type.

The question is, if that would break anything.

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

In here:

http://www.newlisp.org/downloads/develo ... nprogress/

everything is prepared for an extended ffi. Turns out, a new CELL type is not necessary. A 'fcall' is already in primes.h and prototyp for p_fcall in protos.h.

The ->aux field of the import cell which contains the name of the imported function, from now on contains a structure typedef FFIMPORT defined in newlisp.h.

Expand this structure to your liking. If the typefield is 0, newLISP will use the simple interface, else it will call the new interface. All other values are free for you to use. The structure gets allocated during 'import'.

At the end of nl-import.c, I added stubs for:

CELL * executeLibFFI(CELL * pcell, CELL * params)

and

CELL * p_fcall(CELL * params)

Basically everything you do, can be done with those two functions. pcell->aux contains the new structure and pcell->contents contains the imported function address as before. At the moment these functions just return nil. executeLibFFI() is called from executeLibfunction().

For the return values of fcall() and executeLibFFI() look into the stuff......() functions in newlisp.c. They generate new newLISP cells for all the specific types.

The way things are setup, no bad side effects to other portions of newLISP can happen. If you want, you can pull out executeLibFFI() and p_fcall() in extra file.

development newlisp-10.3.6 will be released later today. But you can start with the "inprogess" version, if you want.

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

The way things are setup, no bad side effects to other portions of newLISP can happen. If you want, you can pull out executeLibFFI() and p_fcall() in extra file.

development newlisp-10.3.6 will be released later today. But you can start with the "inprogess" version, if you want.
Somebodys done my homework :-)

Thanks alot.

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

Is there any public cvs/git/svn/fossil/bazzar/mercurial repo ?

Or, do you prefere patches or complete tar files ?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

You can mail me a .tgz of the files changed.

I just posted the 10.3.6 development release in:

http://www.newlisp.org/downloads/development/

you can take this as the starting point. It already has a #define FFI in newlisp.h and some other changes not previously in inprogress/ .

sunmountain
Posts: 39
Joined: Tue Mar 15, 2011 5:11 am

Re: Compile DLL

Post by sunmountain »

Hi Lutz,
your work has laid the foundaition, so I could relatively easlily code a first demo (until now static variable pushing), but now this one here:

Code: Select all

(import "msvcrt" "printf" "cdecl")
(println (fcall printf a b c))
(exit 0)
gives this here:

Code: Select all

$ newlisp fcall.lsp
Hello FFI - here's newLisp (and the answer is 42)
printf
So I'm looking forward on getting it going further :-)

I just extended FFIMPORT (for now)

Code: Select all


#ifdef FFI
typedef struct
    {
    /* original import name of function */
    char * name;
    /* 0 for traditional import calls */
    int type;
    /* holds information for calling */
    ffi_cif cif;
    /* the ABI to use, mostly FFI_DEFAULT_ABI */
    ffi_abi abi;
    /* structure for use with fcall, newLisp -> C */
    struct fcall
        {
        /* number of arguments for foreign function */
        unsigned int nargs;
        /* returned type */
        ffi_type *rtype;
        /* array of argument types */
        ffi_type *atypes[2];
        /* attay of arguments */
        void *avalues[2];
        } fcall;
    /* structure for use with fclos (FFI callback), C -> newLisp */
    struct fclos
        {
        /* holds information for closure */
        ffi_closure *clos;
        /* the newList cell to be eval'd if called */
        CELL *expr;
        } fclos;
    } FFIMPORT;
#endif
and wrote some code (p_fcall):

Code: Select all

ffi = (FFIMPORT *)pCell->aux;

/* setup FFIMPORT structure */

char *fmt = "Hello FFI - here's newLisp (and the answer is %i)\n";
int i = 42;
void *aval[2];
aval[0] = &fmt;
aval[1] = &i;
ffi_type *argt[2];
argt[0] = &ffi_type_pointer;
argt[1] = &ffi_type_uint;
ffi_cif cif;

status = ffi_prep_cif(&cif,FFI_DEFAULT_ABI,2,&ffi_type_uint,argt);
if(status != FFI_OK)
    {
    return stuffString("ffi_prep_cif failed");
    }
else
    {
    ffi_call(&cif,FFI_FN(printf),&result,aval);
    }

return stuffString(ffi->name);
For the fcall stuff, the rest now is correct argument parsing and setup.
It should of course return an executable cell in the end, though, to keep the facility newLisp
has with import.

BTW: What's the C coding style named, you use ? Is this K&R ?

Lutz
Posts: 5289
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California
Contact:

Re: Compile DLL

Post by Lutz »

Not sure, what that style is called. In the beginning 90's I was reading some of Charles Petzold's books about Windows programming. He used to write the same style (not any more today). Many others back then would use this kind of style too. Today another style is in fashion, not indenting the braces but only the code. I like the other one more.

Use whatever you feel comfortable with.

Locked