Page 1 of 1

Why is the behavior of "trim" function so strange?

PostPosted: Tue Aug 15, 2017 12:26 pm
by psilwen
Code: Select all
newLISP v.10.7.1 32-bit on Windows IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str (dup "\000" 10))
"\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 31))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim

Re: Why is the behavior of "trim" function so strange?

PostPosted: Fri Aug 18, 2017 1:33 pm
by varbanov
Hi,

I'm still on a previous version and there's no problem ...
Code: Select all
newLISP v.10.6.2 32-bit on Win32 IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 52))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
>


Yours,
s.v.

Re: Why is the behavior of "trim" function so strange?

PostPosted: Fri Aug 18, 2017 2:01 pm
by CaveGuy
I am back from the future :) no problem there ?

Code: Select all
newLISP v.10.7.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "\000" 52))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
>

Re: Why is the behavior of "trim" function so strange?

PostPosted: Fri Aug 18, 2017 6:03 pm
by psilwen
Code: Select all
newLISP v.10.7.3 32-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 1))
"\000"
> (trim str)
""
> (setq str (dup "\000" 2))
"\000\000"
> (trim str)
""
> (setq str (dup "\000" 3))
"\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 4))
"\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 5))
"\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 6))
"\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 7))
"\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 9))
"\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 10))
"\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 11))
"\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 12))
"\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 13))
"\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 14))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 15))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 16))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 17))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 18))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 19))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 20))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0"
> (trim str)
""
> (setq str (dup "\000" 21))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000"
> (trim str)
""
> (setq str (dup "\000" 22))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000"
> (trim str)
""
> (setq str (dup "\000" 23))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 24))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 25))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 26))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 27))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 28))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 29))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 30))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 31))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 33))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim




Code: Select all
newLISP v.10.7.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 1))
"\000"
> (trim str)
""
> (setq str (dup "\000" 2))
"\000\000"
> (trim str)
""
> (setq str (dup "\000" 3))
"\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 4))
"\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 5))
"\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 6))
"\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 7))
"\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 9))
"\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 10))
"\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 11))
"\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 12))
"\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 13))
"\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 14))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 15))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 16))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 17))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 18))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 19))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 20))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0"
> (trim str)
""
> (setq str (dup "\000" 21))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000"
> (trim str)
""
> (setq str (dup "\000" 22))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000"
> (trim str)
""
> (setq str (dup "\000" 23))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 24))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 25))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 26))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 27))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 28))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 29))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 30))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 31))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 33))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 34))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 35))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 36))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 37))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 38))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 39))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 40))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0"
> (trim str)
""
> (setq str (dup "\000" 41))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000"
> (trim str)
""
> (setq str (dup "\000" 42))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000"
> (trim str)
""

Re: Why is the behavior of "trim" function so strange?

PostPosted: Fri Aug 18, 2017 6:09 pm
by psilwen
newLISP v.10.7.3
newLISP v.10.7.1
newLISP v.10.7.0
All have this issues.

Usually starting from 8, 16, 24, 32...., depending on the version.

But newLISP v.10.6.2,I tested from 1 to 65536, everything was fine.

Re: Why is the behavior of "trim" function so strange?

PostPosted: Sat Aug 19, 2017 6:40 am
by HPW
Hello,

trim seems not to like the null-character.
So is it valid whitespace character?

Code: Select all
newLISP v.10.7.2 32-bit on Windows IPv4/6 libffi, options: newlisp -h
> (setq str "\t\000   \t")
"\t\000   \t"
> (trim str)

ERR: not enough memory in function trim


Code: Select all
> (setq str "\tabc\000   abc\t")
"\tabc\000   abc\t"
> (trim str)
"abc\000   abc"
>


Regards

Re: Why is the behavior of "trim" function so strange?

PostPosted: Sat Aug 19, 2017 4:42 pm
by Lutz
'trim' was overrunning memory when all characters in string < 32 (space).

Fixed here:
http://newlisp.nfshost.com/downloads/de ... nprogress/

Re: Why is the behavior of "trim" function so strange?

PostPosted: Sun Aug 20, 2017 5:41 am
by HPW
Hello Lutz,

Thanks for the fix.
Will characters <32 now handled as whitespace and get trimmed?
Or do they stay?

Regards

Re: Why is the behavior of "trim" function so strange?

PostPosted: Mon Aug 21, 2017 8:01 pm
by Lutz
See last paragraph here:
http://www.newlisp.org/downloads/develo ... 10.7.3.txt

So the 10.7.3 now is more forgiving when a string contains zero characters, which it should'nt in perfect ASCII or UTF-8 where the zero character marks the end of the string.

Re: Why is the behavior of "trim" function so strange?

PostPosted: Wed Sep 13, 2017 5:55 am
by psilwen
trim still has bug.

Code: Select all
newLISP v.10.7.3 32-bit on Windows IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str "\xbb\xe1")
"会"
> (length (trim str))
2
> (length (trim str "\000"))
5
> (setq str "\xce\xaa\xca\xb2\xc3\xb4\xbb\xe1")
"为什么会"
> (length (trim str))
8
> (length (trim str "\000"))
11

Re: Why is the behavior of "trim" function so strange?

PostPosted: Wed Sep 13, 2017 8:20 am
by TedWalther
It occurs to me, if I was trimming a string, and it had \000 characters on either the left or the right side, I'd want them GONE. If it is a binary string where the \000 character belongs, then I wouldn't be running trim on it, since trim is a textual function.

Re: Why is the behavior of "trim" function so strange?

PostPosted: Wed Sep 13, 2017 9:24 am
by ralph.ronnquist
Technically, since neither "\xbb\xe1" nor "\xce\xaa\xca\xb2\xc3\xb4\xbb\xe1" are valid UTF-8 strings, the trim behaviour is conveniently undefined. It appears the trim function expands "\xbb" into a 2-byte code, and "\xe1" into a three byte code, making its output valid UTF-8. ("\xce\xaa", "\xca\xb2" and "\xc3\xb4" are valid UTF-8 characters).