newlispfanclub.alh.net

Hi there,

I'm playing newLisp and having fun. But one thing, there aren't many libraries.

So I think we can make a module for GObject Introspection, then we'll get a lot of libraries out of box (https://live.gnome.org/GObjectIntrospection)

Just copy PyGObject. It's written in C ;-)

I checked, and found sadly, newLISP doesn't support C-made modules (maybe :-( Please tell me if there is)

Is there a way for loading a C-made module? Some features required:

* variables management (including finalizing, need a custom deconstructor)
* custom types
* function calls and code generating (generate wrappers for functions, namespaces and classes types)

Maybe there's something left.. It's not a problem if the C-made module support is full-featured, just like Python.

It seems quite a simple work: if we don't wanna compile our code to executables, just let the interpreter load dynamic libraries after loading the main library.

Unfortunately, newLISP's interpreter is combined with the code which is supposed to be a standalone library... Maybe we need to split them first.

ekd123 wrote:I checked, and found sadly, newLISP doesn't support C-made modules (maybe :-( Please tell me if there is)

Is there a way for loading a C-made module?

Try starting here: http://www.newlisp.org/downloads/newlis ... shared-lib

That's not exactly what I'm talking about. I meant "write module in C and newLisp calls it" not "write a dynamic library in C and newLisp calls it". They're similar but completely different. The first one is inside the Lisp environment and the last one is out of it. We need a lot of work to make the last one work but the first one works out of box. That's the difference.

It's quite tricky to make such a thing in pure Lisp (I was getting so much trouble doing it in Common Lisp). The most problems are around types. Writing it in C is more easier, what's more, faster. The most important thing, we have much less problems. :-D

What do you mean precisely by "and newLISP calls it"? And your continual use of the (industry-abused) word "module" isn't very clear to me either.

Do you mean that "the first one" is the linking of C procedures to newLISP (like the primitives, but users could write them)? I think that's what you meant but I'm not entirely sure.

This would mean, at least at a low level, that you'd have to re-build newLISP (if C procedures are statically linked), or re-build a dynamic link library that the already-built newLISP already references.

I gather from your description that the "module system" you imagine would be designed with a newLISP substrate in mind (similar to the one you did in Common Lisp). But what sort of advantages do you see with the "module system" over using the current import "system" ("the last thing")? That is, is there something disadvantageous or somehow less desirable about using import? Also, would you reveal any more details on the design of the "module system" to give us an idea upon which to evaluate any entailed advantages to using such a system over import? I am very curious, and I think the answer to these types of questions will help Lutz in weighing in on this issue.

Thanks! —Rick

rickyboy wrote:Do you mean that "the first one" is the linking of C procedures to newLISP (like the primitives, but users could write them)? I think that's what you meant but I'm not entirely sure.

Yes, but they're not linked in newLISP. They can be loaded and unloaded easily.

rickyboy wrote:This would mean, at least at a low level, that you'd have to re-build newLISP (if C procedures are statically linked), or re-build a dynamic link library that the already-built newLISP already references.

That's the problem. I've said perhaps we need to split the core and the shell. Just like Python, there is python and libpython.

rickyboy wrote:I gather from your description that the "module system" you imagine would be designed with a newLISP substrate in mind (similar to the one you did in Common Lisp). But what sort of advantages do you see with the "module system" over using the current import "system" ("the last thing")? That is, is there something disadvantageous or somehow less desirable about using import? Also, would you reveal any more details on the design of the "module system" to give us an idea upon which to evaluate any entailed advantages to using such a system over import? I am very curious, and I think the answer to these types of questions will help Lutz in weighing in on this issue.

[/quote]
Here we used some GObject/GObjectIntrospection terms:
(1) a dynamic library which has a GObject class, already registered in GI
(2) use "g_irepository_require" to load it.
(3) make a wrapper for all the functions(the problem), and the class type
(4) when we call the function wrapper, it calls "g_function_info_invoke" to invoke the original function. Two arguments' types are both "GIArgument", it's a union in C. Handle it in Lisp is quite tricky. In SBCL, it may cause crashes with no reasons... :-(
(5) when an object is no longer needed, we need to call "g_object_unref" to decrease the reference count.
(6) namespaces for each namespace in GObjectIntrospection Repository

It's possible to do it in pure Lisp, but tricky. It's much easier to put it in C ;-)

Also, "modules" extends Lisp directly, unlike "dynamic libraries".

I borrowed the term "module" from http://www.newlisp.org/modules/, though it may be confusing.

Instead of doing this:

ekd123 wrote:Here we used some GObject/GObjectIntrospection terms:
(1) a dynamic library which has a GObject class, already registered in GI
(2) use "g_irepository_require" to load it.
(3) make a wrapper for all the functions(the problem), and the class type
(4) when we call the function wrapper, it calls "g_function_info_invoke" to invoke the original function. Two arguments' types are both "GIArgument", it's a union in C. Handle it in Lisp is quite tricky. In SBCL, it may cause crashes with no reasons... :-(
(5) when an object is no longer needed, we need to call "g_object_unref" to decrease the reference count.
(6) namespaces for each namespace in GObjectIntrospection Repository

Why not do this simpler thing?:

1. Code your library in C
2. Compile it into a dynamic library
3. Use import to access the interface
:-)

Call me dumb or ignorant, but I guess I still don't see the advantage to using your "module system" (based on the ideas of GI, which up until now I only thought stood for "Gastrointestinal" :) over dynamic libraries/import. What are the advantages? Maybe presenting a good use case might help.

Well, that's not the advantage of "c-made module" itself, but GObjectIntrospection's.

Making a lot of bindings by hand is annoying. in this case, GI helps us out. You could check out the homepage of GI to find out how great it is.

Well, I thought you'd make a case for it, but instead you're telling me to "go hunt the rabbit." No, but thanks anyway for the offer.

Looks like, then, it is time for you to code, my friend. :) Good luck and let us know how it goes!

Not sure what your "case" means? ...

In my opionion, Lisp is powerful enough to do such thing. But if there's some thing more low-level, it's easier. That's the only advantage. :-/

For example, here is a function called "H", which loads a metadata file (filled with function info of library L), loads L and generates a lot of function wrappers.. like this:
In this way, the library developer writes a metadata, we just use this and can "import" functions. We don't need to import every function by hand any more. In pure Lisp, the "dispatcher" can't be implemented (maybe?) (disadvantages: waste of memory, start slowly)

Not sure what your "case" means? ...

It meant "argument", in that context.

BTW, thank you for explaining this via the pseudo/stub code snippet. It made it easier for me to understand than having to take the time to read about it in the documentation of a system I know *nothing* about -- I would have had to first read about the context around GI, before reading the GI stuff which was too terse for me anyway. (That was the "hunting the rabbit" part I talked about. :) On then to matters of direct consideration.

First, I'm sure we agree that library developers foreign to newLISP are not going to code their libraries this way (unless I'm unaware that this is some universally adopted convention or standard). So that means that, in essence, newLISP developers will be the library writers.

Second, I take it from your code example (middleware?) that this "layer" is boilerplate code that takes care of registering the signatures of the library interface into newLISP. Is that right?

If the above were all true, and your system could be coded with no inherent breakage (I haven't thought about this too much), then two questions remain for me.

The first is: what is the difference of the level of newLISP developer effort between writing the metadata file and on the other hand writing an import-based interface?

If the level of effort is clearly lower in your system (and to boot less error-prone), then your system would be advantageous to use; otherwise, not. You had said that "the library developer writes a metadata" which leads me to believe that the metadata is coded "by hand" (the words which you used to intimate that this, hand coding, is an undesirable part of using the import scheme). On the other hand, if your system could handle an automatic writing of such a metadata file, thereby obviating the step of the developer hand coding the metadata, then your system would clearly be superior to the import system (which requires the hand coding of the newLISP interface (i.e. import calls and friends)).

The next question is: would your system be more runtime efficient than using an import interface? (This goes to your "waste of memory, start slowly" comment, and I treat here as independent of the developer effort (time) issue which was addressed above.) If your system can be demonstrated to be more efficient in time and space at runtime than using an import interface, then on this issue, your system would clearly be advantageous; otherwise, not.

In my view, I believe you are at a point where the only way to properly answer these questions with surety, is to code such a system as you describe, and demonstrate its advantageousness by profiing metrics (e.g. measure developer time to code the metadata and any glue code (but not middleware boilerplate) versus coding an import interface, and measure run time and space for loading and running such facilities as are provided by the (newLISP developer) library under your system versus measuring same for import interface loads and executions of an equivalent library coded the traditional way).

By the way, another interesting question, albeit not material to the "goodness" of your system per se, is: will most newLISP programmers care? If they mostly care about only programming in newLISP, and have no need or desire to program in other languages like C, then that might affect the newLISP developer's choice of adopting your system into newLISP. This is not as important as the other questions — it's certainly not a technical issue, but it is a social issue (at least) and one that I think is interesting.

Best, —Rick

>> First, I'm sure we agree that library developers foreign to newLISP are not going to code their libraries this way (unless I'm unaware that this is some universally adopted convention or standard). So that means that, in essence, newLISP developers will be the library writers.

They can do it when they want =)

>> Second, I take it from your code example (middleware?) that this "layer" is boilerplate code that takes care of registering the signatures of the library interface into newLISP. Is that right?

Yes.

>> The first is: what is the difference of the level of newLISP developer effort between writing the metadata file and on the other hand writing an import-based interface?

import needs library users do something for it, it's just opposite if library devs write a "metadata".

>> On the other hand, if your system could handle an automatic writing of such a metadata file, thereby obviating the step of the developer hand coding the metadata, then your system would clearly be superior to the import system (which requires the hand coding of the newLISP interface (i.e. import calls and friends)).

Yes. There's an automatic generator.

>> The next question is: would your system be more runtime efficient than using an import interface? (This goes to your "waste of memory, start slowly" comment, and I treat here as independent of the developer effort (time) issue which was addressed above.) If your system can be demonstrated to be more efficient in time and space at runtime than using an import interface, then on this issue, your system would clearly be advantageous; otherwise, not.

Sorry for the ambiguity. I meant "if the dispatcher can't be implemented, it wastes memory and takes much time to generate wrappers for foreign function".

For instance:
vs.
>> In my view, I believe you are at a point where the only way to properly answer these questions with surety, is to code such a system as you describe, and demonstrate its advantageousness by profiing metrics (e.g. measure developer time to code the metadata and any glue code (but not middleware boilerplate) versus coding an import interface, and measure run time and space for loading and running such facilities as are provided by the (newLISP developer) library under your system versus measuring same for import interface loads and executions of an equivalent library coded the traditional way).

Comparing with import, running speed will be even slower. Here's an FFI layer and we need to use it for arg type checking automatically. I only meant "starting slowly". It takes time for evaling.

>> will most newLISP programmers care?

At least UNIX developers care. A free UNIX desktop environment, GNOME, is fully written in C and it has a great collection of open-source libraries that all provide "metadata" out of box. However, though not most newLISP programmers, but it may make people see newLISP and give it a try. Then, I would use it for most of my work. =) It has everything that I need.

Any activities on this?

C-API for extensibility is really a neccessary feature for modern scripting :-O