Run newlisp like multiple instance in one dll and debug in a

[iNPRwANG]

Hi everyone!
I'm a Chinese and with a little English : ) and this is my first post.
I used newlisp period of time, I think it's a very good and easy to use script as a lisp language.

One day, I want to use it in a Microsoft Windows GUI application, I use the export function "newlispEvalStr" to eval script, It looks worked good. But I can not use the "debug" function to step run a newlisp route, then I add the function in source code:

Code: Select all

void EXPORT AllocDebugConsole() {

	IOchannel = stdin;

	AllocConsole();
	freopen("CONOUT$","w+t",stdout);
	freopen("CONIN$","r+t",stdin);  
}

And I invoked it in the GUI application intialize time. The console window had be showed, and the debug features of newlisp worked!


The second problem:

Sometimes I want to use the newlisp with multiple thread or mutiple instance.
Because of the GUI application could not block the user input route at any time, and the user input could fire the multiple things to do, if I want to do these things with newlisp script, I must use the newlisp as multi-thread environment.

I'd read the source code of newlisp, the "newlispEvalStr" route is not thread safe, this means it could not run at the same time because of newlisp use many global variables like cellCount, symbolCount, envStack, resultStack and so on.

Then, I modified the source code: collect the global variables with a ENVCONTEXT, and modified all the functions with a first argument, just as the ENVCONTEXT,like this:

Code: Select all

typedef struct tagEnvContext
{
	char startupDir[PATH_MAX]; /* start up directory, if defined via -w */	
	FILE * IOchannel;          
	int ADDR_FAMILY;		
	int IOchannelIsSocket;	
	int MAX_CPU_STACK;
	long MAX_CELL_COUNT;
	size_t cellCount;
	size_t symbolCount;
	int recursionCount;
	UINT printDevice;
	UINT * resultStack;		
	UINT * resultStackIdx;	
	UINT * envStack;        
	UINT * envStackIdx;	
	CELL * trueCell;		
	CELL * nilCell;		
	STREAM strStream;
	SYMBOL * nilSymbol;
	SYMBOL * trueSymbol;
	SYMBOL * startSymbol;
	SYMBOL * questionSymbol;
	SYMBOL * atSymbol;		
	SYMBOL * mainContext;	
	SYMBOL * currentContext;
	SYMBOL * errorEvent;	
	SYMBOL * symbolCheck;
	SYMBOL * itSymbol;		
	void * stringIndexPtr;	
	CELL * stringCell;		
	int traceFlag;			
	int errorReg;	
	jmp_buf errorJump;
	int pushResultFlag;
	int prettyPrintFlags;
	char lc_decimal_point;
	CELL * xmlTags;			
	CELL * xmlCallback;		
	CELL * (*evalFunc)(void * pContext, CELL *);
	CELL * firstFreeCell;	
	CELL * lastCellCopied;	
	SYMBOL * plusSymbol;
	SYMBOL * currentFunc;	
	SYMBOL * argsSymbol;	
	SYMBOL * mainArgsSymbol;
	SYMBOL * dolistIdxSymbol;
	SYMBOL * sysSymbol[MAX_REGEX_EXP];
	SYMBOL * timerEvent;
	SYMBOL * promptEvent;
	SYMBOL * commandEvent;
	SYMBOL * transferEvent;
	SYMBOL * readerEvent;	
	SYMBOL * symHandler[32];
	int currentSignal;
	CELL * throwResult;
	STREAM readLineStream;
	STREAM errorStream;
	int parStackCounter;
	UINT * envStackTop;
	UINT * resultStackTop;
	UINT * lambdaStack;
	UINT * lambdaStackIdx;
	SYMBOL objSymbol;	
	CELL * objCell;
	int evalSilent;
	int evalCatchFlag;
	int prettyPrintPars;
	int prettyPrintCurrent;
	int prettyPrintLength;
	char * prettyPrintTab;
	char * prettyPrintFloat;
	UINT prettyPrintMaxLength;
	int stringOutputRaw;
	char logFile[PATH_MAX];
	int pagesize;
	int logTraffic;
	int connectionTimeout;
	char * IOdomain;	
	int IOport;
	int IOchannelIsSocketStream;
	int isTTY;
	int demonMode;
	int noPromptMode;
	int forcePromptMode;
	int httpMode;
	CELL * cellMemory;/* start of cell memory */
	CELL * cellBlock; /* the last block allocated */	
	int dllInitialized;	
	STREAM libStrStream;
}ENVCONTEXT, * PENVCONTEXT;

And functions are modified like this: (this is the example, and I'd modified all of the functions)


The old:

Code: Select all

void initStacks(PENVCONTEXT pContext)
{
MAX_ENV_STACK = MAX_CPU_STACK * 8 * 2;
MAX_RESULT_STACK = MAX_CPU_STACK * 2;
if(envStack != NULL) freeMemory(envStack);
if(resultStack != NULL) freeMemory(resultStack);
if(lambdaStack != NULL) freeMemory(lambdaStack);
envStackIdx = envStack = (UINT *)allocMemory((MAX_ENV_STACK + 16) * sizeof(UINT));
envStackTop = envStack + MAX_ENV_STACK;
resultStackIdx = resultStack = (UINT *)allocMemory((MAX_RESULT_STACK + 16) * sizeof(UINT));
resultStackTop = resultStack + MAX_RESULT_STACK;
lambdaStackIdx = lambdaStack = (UINT *)allocMemory((MAX_RESULT_STACK + 16) * sizeof(UINT));
}

And the new:

Code: Select all

void initStacks(PENVCONTEXT pContext)
{
MAX_ENV_STACK = (pContext->MAX_CPU_STACK * 8 * 2);
MAX_RESULT_STACK = (pContext->MAX_CPU_STACK * 2);
if(pContext->envStack != NULL) freeMemory(pContext->envStack);
if(pContext->resultStack != NULL) freeMemory(pContext->resultStack);
if(pContext->lambdaStack != NULL) freeMemory(pContext->lambdaStack);
pContext->envStackIdx = pContext->envStack = (UINT *)allocMemory(pContext, (MAX_ENV_STACK + 16) * sizeof(UINT));
pContext->envStackTop = pContext->envStack + MAX_ENV_STACK;
pContext->resultStackIdx = pContext->resultStack = (UINT *)allocMemory(pContext, (MAX_RESULT_STACK + 16) * sizeof(UINT));
pContext->resultStackTop = pContext->resultStack + MAX_RESULT_STACK;
pContext->lambdaStackIdx = pContext->lambdaStack = (UINT *)allocMemory(pContext, (MAX_RESULT_STACK + 16) * sizeof(UINT));
}

And I export a rote named "CreateContext" to create the new newlisp environment context:

Code: Select all

PENVCONTEXT EXPORT CreateContext() {
	return (PENVCONTEXT)malloc(sizeof(ENVCONTEXT));
}

After these changes, I could use the code under C to create multiple environment context and run these in difference theads:

Code: Select all

PENVCONTEXT p_contextA = CreateContext();
PENVCONTEXT p_contextB = CreateContext();


and in thread A
newlispEvalStr(p_contextA, "codes...");

and in thread B
newlispEvalStr(p_contextB, "codes...");

If I want to change data with threads, I may use the share memory.




Then, I have another thinking: the newlisp had a "import" route, it can import the export function from a module, but the GUI application of mine is a single executable file, I can not use the "import".

In order to resolve the problem, I just add a "importFunc" route in the source code:

Code: Select all

int EXPORT importFunc(const char * szFunName, UINT ufuncAddr, DWORD dwCallType)
{
    SYMBOL * symbol;
    CELL * pCell;
    
    if (dwCallType == 0)
        pCell = getCell(pContext, CELL_IMPORT_DLL);    
    else if (dwCallType == 1)
        pCell = getCell(pContext, CELL_IMPORT_CDECL);
        
    symbol = translateCreateSymbol(pContext, szFunName, pCell->type, pContext->currentContext, TRUE);
    if(isProtected(symbol->flags))
        return(errorProcExt2(pContext, ERR_SYMBOL_PROTECTED, stuffSymbol(pContext, symbol)));
	
    deleteList(pContext, (CELL *)symbol->contents);
    symbol->contents = (UINT)pCell;
    pCell->contents = ufuncAddr;
    pCell->aux = (UINT)symbol->name;
    
    if(pCell->contents == 0)
        return 0;
    
    return 1;
}

The first argument "szFunName" has specify the name of the rote to import to newlisp, the second argument "ufuncAddr" has specify the function address to import. The invoke example in C code like this:

//fun1 is a C rote with stdcall type
importFunc("fun1", fun1, 0);

It works good.

[Lutz]

Your AllocDebugConsole() function is a nice idea and I will add it into the next version of the newLISP DLL.

Rather then running multiple instances of newLISP in the same DLL, I would run multiple instances of newlisp.exe and import Windows GUI functions using 'import' and do callbacks using 'callback'. There is a good example of this in newlisp-x.x.x/examples/win32demo.lsp. This demo was written by Cyril Slobin and you can also access it at:

http://slobin.pp.ru/newlisp/win32demo.lsp

On multicore CPUs this solution will scale much better than multiple threads in the same DLL, because the OS will distribute the different newlisp.exe processes on to different cores and you have a solution working on Windows and UNIX at the same time.

Welcome to newLISP :)

[iNPRwANG]

Your AllocDebugConsole() function is a nice idea and I will add it into the next version of the newLISP DLL.

I'm very glad to hear that. : )

Rather then running multiple instances of newLISP in the same DLL, I would run multiple instances of newlisp.exe and import Windows GUI functions using 'import' and do callbacks using 'callback'. There is a good example of this in newlisp-x.x.x/examples/win32demo.lsp. This demo was written by Cyril Slobin and you can also access it at:

http://slobin.pp.ru/newlisp/win32demo.lsp

On multicore CPUs this solution will scale much better than multiple threads in the same DLL, because the OS will distribute the different newlisp.exe processes on to different cores and you have a solution working on Windows and UNIX at the same time.

Thanks for your suggestion.
I think multiple instance of newlisp.exe to build a server side program is the nice way, for example: use a listener(dispatcher) instance and multiple instance to run actual things.


Back to the topic, if I use your solution in a GUI application formally, some problems are appear:

1. For people of China, most of them enjoy GUI applications generally, and most people use the Microsoft Windows. A multiple instance GUI program would puzzling them, for example: they will consider the program as virus. : (

2. While the GUI main program crash, multiple child instance are hard to be terminate. (Users were stupid generally, while the single executable application carshed, they could restart it simply.)

3. Of the exist GUI program, I had built it for a long time, with many C++ wrapper GUI class. I tired to rebuild it with full newlisp script.



So, I use the way of my posted, create multiple ENVCONTEXT to solution my problems, I think it's has these merits:

1. A easy way to resolve the above problems.

2. Make data exchange easier.

For example 1:

In multiple thread mode, I can exchange data between multiple ENVCONTEXT simply as use same synchronization methods.

For example 2:

I can use the way to make the separate newlisp environment startup faster, if I want to startup many newlisp instance, and each of them must load the startup script, while the script be larger, multiple instance will cost a long time.

In my solution, I can create a init ENVCONTEXT and let it to load the startup script, the second ENVCONTEXT I let it simply clone the init ENVCONTEXT's data, and the third ENVCONTEXT clone the data from init ENVCONTEXT also ...



I think onn multicore CPUs my solution will also effective, the multiple threads on Windows can be distributed to different cores as well as multiple process.


I am not sure if I am correct. : )