*** glibc detected *** double free or corruption (!prev): 0x

[newdep]

yes thats correct I have everytime different chracters just befor [/text]
sometimes highasci sometimes other things..
I cant explain that... a LOCALE issue?

[pjot]

Same problems occur when I compile newLisp WITHOUT support for READLINE.

Peter

[pjot]

Using Valgrind:

peter@Solarstriker:~$ valgrind --leak-check=full newlisp
==28837== Memcheck, a memory error detector.
==28837== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==28837== Using LibVEX rev 1471, a library for dynamic binary translation.
==28837== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==28837== Using valgrind-3.1.0, a dynamic binary instrumentation framework.
==28837== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==28837== For more details, rerun with: -v
==28837==
newLISP v.8.7.6 on linux, execute 'newlisp -h' for more info.

> (setq url (get-url "http://www.newlisp.org/rss.cgi?News"))
==28837== Invalid write of size 1
==28837== at 0x8062F63: getPutPostUrl (nl-web.c:525)
==28837== by 0x80625E0: p_getUrl (nl-web.c:179)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804F642: setDefine (newlisp.c:3597)
==28837== by 0x804F486: p_setq (newlisp.c:3516)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804AECB: evaluateStream (newlisp.c:855)
==28837== by 0x804AD19: executeCommandLine (newlisp.c:833)
==28837== by 0x804A645: main (newlisp.c:686)
==28837== Address 0x41D7FAC is 0 bytes after a block of size 2,324 alloc'd
==28837== at 0x401C811: realloc (vg_replace_malloc.c:306)
==28837== by 0x806312B: getPutPostUrl (nl-web.c:482)
==28837== by 0x80625E0: p_getUrl (nl-web.c:179)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804F642: setDefine (newlisp.c:3597)
==28837== by 0x804F486: p_setq (newlisp.c:3516)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804AECB: evaluateStream (newlisp.c:855)
==28837== by 0x804AD19: executeCommandLine (newlisp.c:833)
==28837== by 0x804A645: main (newlisp.c:686)
==28837==
==28837== Invalid read of size 1
==28837== at 0x401D500: memcpy (mac_replace_strmem.c:394)
==28837== by 0x804BF74: copyCell (newlisp.c:1485)
==28837== by 0x804F64A: setDefine (newlisp.c:3597)
==28837== by 0x804F486: p_setq (newlisp.c:3516)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804AECB: evaluateStream (newlisp.c:855)
==28837== by 0x804AD19: executeCommandLine (newlisp.c:833)
==28837== by 0x804A645: main (newlisp.c:686)
==28837== Address 0x41D7FAC is 0 bytes after a block of size 2,324 alloc'd
==28837== at 0x401C811: realloc (vg_replace_malloc.c:306)
==28837== by 0x806312B: getPutPostUrl (nl-web.c:482)
==28837== by 0x80625E0: p_getUrl (nl-web.c:179)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804F642: setDefine (newlisp.c:3597)
==28837== by 0x804F486: p_setq (newlisp.c:3516)
==28837== by 0x804B681: evaluateExpression (newlisp.c:1058)
==28837== by 0x804AECB: evaluateStream (newlisp.c:855)
==28837== by 0x804AD19: executeCommandLine (newlisp.c:833)
==28837== by 0x804A645: main (newlisp.c:686)

I changed line 525 in 'nl-web.c' to:

*(resultPtr + resultSize - 1) = 0;


Now it works.

Peter

[pjot]

I still have the invalid read though:

newLISP v.8.7.6 on linux, execute 'newlisp -h' for more info.

> (setq url (get-url "http://www.newlisp.org/rss.cgi?News"))
==29098== Invalid read of size 1
==29098== at 0x401D500: memcpy (mac_replace_strmem.c:394)
==29098== by 0x804C124: copyCell (newlisp.c:1485)
==29098== by 0x804F7FA: setDefine (newlisp.c:3597)
==29098== by 0x804F636: p_setq (newlisp.c:3516)
==29098== by 0x804B831: evaluateExpression (newlisp.c:1058)
==29098== by 0x804B07B: evaluateStream (newlisp.c:855)
==29098== by 0x804AEC9: executeCommandLine (newlisp.c:833)
==29098== by 0x804A800: main (newlisp.c:663)
==29098== Address 0x4232CC4 is 0 bytes after a block of size 2,324 alloc'd
==29098== at 0x401C811: realloc (vg_replace_malloc.c:306)
==29098== by 0x80632DB: getPutPostUrl (nl-web.c:482)
==29098== by 0x8062790: p_getUrl (nl-web.c:179)
==29098== by 0x804B831: evaluateExpression (newlisp.c:1058)
==29098== by 0x804F7F2: setDefine (newlisp.c:3597)
==29098== by 0x804F636: p_setq (newlisp.c:3516)
==29098== by 0x804B831: evaluateExpression (newlisp.c:1058)
==29098== by 0x804B07B: evaluateStream (newlisp.c:855)
==29098== by 0x804AEC9: executeCommandLine (newlisp.c:833)
==29098== by 0x804A800: main (newlisp.c:663)

This seems to be caused by the fact that this line in newlisp.c (1485);

memcpy((void *)newCell->contents,(void*)cell->contents, (UINT)cell->aux);


...copies 1 byte too much. So the (UINT)cell-aux is not correct.

Peter

[Lutz]

Can you check if you can see the same thing in 8.7.1? 'get-url' had changes on 8.7.2/3.

Thanks

Lutz

[Lutz]

... and does this happend only with "chunked" pages? You can check this when retrieving with the "header" option and it says "Transfer-Encoding: chunked".

I would do all this by myself, but the problem just doesn't want to show up on other OSs than Slackware.

Lutz

[pjot]

Yes it happens with newLisp 8.7.1 as well, even more:

peter@Starcrater:~$ valgrind --leak-check=full newlisp
==9346== Memcheck, a memory error detector.
==9346== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==9346== Using LibVEX rev 1471, a library for dynamic binary translation.
==9346== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==9346== Using valgrind-3.1.0, a dynamic binary instrumentation framework.
==9346== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==9346== For more details, rerun with: -v
==9346==
==9346== Conditional jump or move depends on uninitialised value(s)
==9346== at 0x804DABB: compileExpression (newlisp.c:2528)
==9346== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9346== by 0x804AF6A: evaluateStream (newlisp.c:846)
==9346== by 0x804D7EB: loadFile (newlisp.c:2437)
==9346== by 0x804A3E7: loadStartup (newlisp.c:430)
==9346== by 0x804A796: main (newlisp.c:554)
==9346==
==9346== Conditional jump or move depends on uninitialised value(s)
==9346== at 0x804DABB: compileExpression (newlisp.c:2528)
==9346== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9346== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9346== by 0x804AF6A: evaluateStream (newlisp.c:846)
==9346== by 0x804D7EB: loadFile (newlisp.c:2437)
==9346== by 0x804A3E7: loadStartup (newlisp.c:430)
==9346== by 0x804A796: main (newlisp.c:554)
==9346==
==9346== Conditional jump or move depends on uninitialised value(s)
==9346== at 0x804DABB: compileExpression (newlisp.c:2528)
==9346== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9346== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9346== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9346== by 0x804AF6A: evaluateStream (newlisp.c:846)
==9346== by 0x804D7EB: loadFile (newlisp.c:2437)
==9346== by 0x804A3E7: loadStartup (newlisp.c:430)
==9346== by 0x804A796: main (newlisp.c:554)
newLISP v.8.7.1 on linux, execute 'newlisp -h' for more info.

> (setq url (get-url "http://www.newlisp.org/rss.cgi?News"))
==9346==
==9346== Invalid write of size 1
==9346== at 0x8062530: getPutPostUrl (nl-web.c:529)
==9346== by 0x8061CA0: p_getUrl (nl-web.c:183)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804F6F2: setDefine (newlisp.c:3582)
==9346== by 0x804F536: p_setq (newlisp.c:3501)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804AFAB: evaluateStream (newlisp.c:850)
==9346== by 0x804ADF9: executeCommandLine (newlisp.c:828)
==9346== by 0x804A73B: main (newlisp.c:658)
==9346== Address 0x4260BE4 is 0 bytes after a block of size 2,324 alloc'd
==9346== at 0x401C811: realloc (vg_replace_malloc.c:306)
==9346== by 0x80626CA: getPutPostUrl (nl-web.c:486)
==9346== by 0x8061CA0: p_getUrl (nl-web.c:183)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804F6F2: setDefine (newlisp.c:3582)
==9346== by 0x804F536: p_setq (newlisp.c:3501)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804AFAB: evaluateStream (newlisp.c:850)
==9346== by 0x804ADF9: executeCommandLine (newlisp.c:828)
==9346== by 0x804A73B: main (newlisp.c:658)
==9346==
==9346== Invalid read of size 1
==9346== at 0x401D500: memcpy (mac_replace_strmem.c:394)
==9346== by 0x804C064: copyCell (newlisp.c:1476)
==9346== by 0x804F6FA: setDefine (newlisp.c:3582)
==9346== by 0x804F536: p_setq (newlisp.c:3501)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804AFAB: evaluateStream (newlisp.c:850)
==9346== by 0x804ADF9: executeCommandLine (newlisp.c:828)
==9346== by 0x804A73B: main (newlisp.c:658)
==9346== Address 0x4260BE4 is 0 bytes after a block of size 2,324 alloc'd
==9346== at 0x401C811: realloc (vg_replace_malloc.c:306)
==9346== by 0x80626CA: getPutPostUrl (nl-web.c:486)
==9346== by 0x8061CA0: p_getUrl (nl-web.c:183)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804F6F2: setDefine (newlisp.c:3582)
==9346== by 0x804F536: p_setq (newlisp.c:3501)
==9346== by 0x804B76F: evaluateExpression (newlisp.c:1018)
==9346== by 0x804AFAB: evaluateStream (newlisp.c:850)
==9346== by 0x804ADF9: executeCommandLine (newlisp.c:828)
==9346== by 0x804A73B: main (newlisp.c:658)

Same problem, line 529 says:

Code: Select all

*(resultPtr + resultSize) = 0;

If I add the '-1' the problem is gone. I'll check the 'chunked' issue now.

Peter

[pjot]

I seem not be able to find a chunked' page. Some other thing is interesting though:

peter@Starcrater:~$ valgrind --leak-check=full newlisp
==9606== Memcheck, a memory error detector.
==9606== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==9606== Using LibVEX rev 1471, a library for dynamic binary translation.
==9606== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==9606== Using valgrind-3.1.0, a dynamic binary instrumentation framework.
==9606== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==9606== For more details, rerun with: -v
==9606==
==9606== Conditional jump or move depends on uninitialised value(s)
==9606== at 0x804DABB: compileExpression (newlisp.c:2528)
==9606== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9606== by 0x804AF6A: evaluateStream (newlisp.c:846)
==9606== by 0x804D7EB: loadFile (newlisp.c:2437)
==9606== by 0x804A3E7: loadStartup (newlisp.c:430)
==9606== by 0x804A796: main (newlisp.c:554)
==9606==
==9606== Conditional jump or move depends on uninitialised value(s)
==9606== at 0x804DABB: compileExpression (newlisp.c:2528)
==9606== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9606== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9606== by 0x804AF6A: evaluateStream (newlisp.c:846)
==9606== by 0x804D7EB: loadFile (newlisp.c:2437)
==9606== by 0x804A3E7: loadStartup (newlisp.c:430)
==9606== by 0x804A796: main (newlisp.c:554)
==9606==
==9606== Conditional jump or move depends on uninitialised value(s)
==9606== at 0x804DABB: compileExpression (newlisp.c:2528)
==9606== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9606== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9606== by 0x804DCF4: compileExpression (newlisp.c:2596)
==9606== by 0x804AF6A: evaluateStream (newlisp.c:846)
==9606== by 0x804D7EB: loadFile (newlisp.c:2437)
==9606== by 0x804A3E7: loadStartup (newlisp.c:430)
==9606== by 0x804A796: main (newlisp.c:554)
newLISP v.8.7.1 on linux, execute 'newlisp -h' for more info.

> (setq url (get-url "http://www.newlisp.org/rss.cgi?News" "header"))
"Date: Sat, 07 Jan 2006 19:18:38 GMT\r\nServer: Apache/1.3.29 (Unix) mod_python/2.7.10 Python/2.2.2 mod_webapp/1.2.0-dev mod_perl/1.29 mod_throttle/3.1.2 PHP/4.3.4\r\nConnection: close\r\nContent-Type: text/xml\r\n\r\n"

So a 'get-url' with the "header" option shows no memory errors. If I immediately after that perform the get-url WITHOUT the "header" option, I receive the same error again.

Peter

[pjot]

Also retrieving 'normal' webpages (not RSS) show the error of reading and writing 1 byte too much.

I am at my 2nd machine now, also with Slackware 10.2 fresh install.

Maybe you could try to install valgrind yourself to see if it happens also? Sometimes there is no crash so maybe that's why you don't see it?. But Valgrind always show the problem of the extra byte. It should show up on Mandrake and other Linux versions also.

It's free software from www.valgrind.org

Peter

[newdep]

Hi Lutz,

The error message is double free, that means a pointer problem...
It's only this get-url issue, the rest works fine...

Also the garbage inside the output befor [/text] indicates some
strange memory mixup...

Slackware uses pure 100% kernel en library stuff no adjustements,
I changed readlib and ncurses and still exists...tested with console/xterm
other teminals but the problems resists in newlisp get-url.

What about the freshmeat compiler farm? do they have slack 10.2?
And can you reproduce it?

Norman.

[Lutz]

Thanks for all the printouts from valgrind, I think I found the problem.

Also reviewing the code I think that all of the "Conditional jump or move depends on uninitialised value(s)" messages do not reveal a problem. valgrind seems to flag all instances where pointers to uninitialized memory are passed, which is ok when that memory is written by the function receiving the pointer.

Unfortunately the only Linux machines I have to test on, are at sourceforge and behind a firewall without web-access for 'get-url'. If it is Ok, I will send you the next development release before publishing it for checking again with valgrind.

Lutz

[pjot]

Fine, no problem!

Peter

[Lutz]

In version 8.7.6 in file nl-web.c line 482 change:

resultPtr = realloc(resultPtr, resultSize + size);

to

resultPtr = realloc(resultPtr, resultSize + size + 1);

This will solve the problem, the "Transfer-Encoding: chunked" option only shows up with the "debug" option:

(get-url "http://www.newlisp.org/rss.cgi?News" "debug")

I will release 8.7.7 this weekend with 'get-url' fix. This version contains also regular expression support for 'directory'

Lutz

ps: thanks to you and Norman for helping to debug this

[pjot]

Results:

peter@Starcrater:~$ valgrind --leak-check=full newlisp
==16488== Memcheck, a memory error detector.
==16488== Copyright (C) 2002-2005, and GNU GPL'd, by Julian Seward et al.
==16488== Using LibVEX rev 1471, a library for dynamic binary translation.
==16488== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==16488== Using valgrind-3.1.0, a dynamic binary instrumentation framework.
==16488== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==16488== For more details, rerun with: -v
==16488==
newLISP v.8.7.6 on linux, execute 'newlisp -h' for more info.

> (setq url (get-url "http://www.newlisp.org/rss.cgi?News"))
[text]<?xml version="1.0" encoding="UTF-8"?>
<!-- generator="newLISP Wiki/CMS rss.cgi v. 1.0" -->
<rss version="0.92">
<channel>
<title>newLISP</title>
<description>News and Comments</description>
<link>http://newlisp.org</link>
<copyright>copyright (c) Lutz Mueller, 2005. All rights reserved.</copyright>
<item>
<title>development version newLISP 8.7.6</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :52%202006 </link>
</item>
<item>
<title>development version newLISP 8.7.5</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :58%202005 </link>
</item>
<item>
<title>development version newLISP 8.7.4</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :18%202005 </link>
</item>
<item>
<title>newLISP coding competition</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=lutz% ... :41%202005 </link>
</item>
<item>
<title>Users about newLISP</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :41%202005 </link>
</item>
<item>
<title>development version newLISP 8.7.3</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :06%202005 </link>
</item>
<item>
<title>repost development version newLISP 8.7.2</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :18%202005 </link>
</item>
<item>
<title>development version newLISP 8.7.2</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=Lutz% ... :14%202005 </link>
</item>
<item>
<title>newLISP release version 8.7.1</title>
<description></description>
<link>http://newlisp.org/index.cgi?find=lutz% ... :50%202005 </link>
</item>
</channel>
</rss>
[/text]

Conclusion: RUNS LIKE A DREAM!

Also it amazes me that the errors during startup of newLisp are gone. :-)

Thanks!
Peter

[newdep]

That solved the problem thanks!

Still its strange that other linux version dont effect this problem
(probably the default handling of malloc_check)

could be quiet dangerous in some situation just to miss1 byte ;-)

Thanks!

Norman.