Why is the behavior of "trim" function so strange?

Q&A's, tips, howto's

Why is the behavior of "trim" function so strange?

Postby psilwen » Tue Aug 15, 2017 12:26 pm

Code: Select all
newLISP v.10.7.1 32-bit on Windows IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str (dup "\000" 10))
"\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 31))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
(reverse "newlisp")
psilwen
 
Posts: 20
Joined: Thu Jul 03, 2014 5:25 am

Re: Why is the behavior of "trim" function so strange?

Postby varbanov » Fri Aug 18, 2017 1:33 pm

Hi,

I'm still on a previous version and there's no problem ...
Code: Select all
newLISP v.10.6.2 32-bit on Win32 IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 52))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
>


Yours,
s.v.
varbanov
 
Posts: 5
Joined: Mon Jul 01, 2013 1:33 pm
Location: Sofia, Bulgaria

Re: Why is the behavior of "trim" function so strange?

Postby CaveGuy » Fri Aug 18, 2017 2:01 pm

I am back from the future :) no problem there ?

Code: Select all
newLISP v.10.7.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "\000" 52))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
>
Bob the Caveguy aka Lord High Fixer.
CaveGuy
 
Posts: 111
Joined: Sun Oct 13, 2002 3:00 pm
Location: Columbus Ohio

Re: Why is the behavior of "trim" function so strange?

Postby psilwen » Fri Aug 18, 2017 6:03 pm

Code: Select all
newLISP v.10.7.3 32-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 1))
"\000"
> (trim str)
""
> (setq str (dup "\000" 2))
"\000\000"
> (trim str)
""
> (setq str (dup "\000" 3))
"\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 4))
"\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 5))
"\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 6))
"\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 7))
"\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 9))
"\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 10))
"\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 11))
"\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 12))
"\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 13))
"\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 14))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 15))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 16))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 17))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 18))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 19))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 20))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0"
> (trim str)
""
> (setq str (dup "\000" 21))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000"
> (trim str)
""
> (setq str (dup "\000" 22))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000"
> (trim str)
""
> (setq str (dup "\000" 23))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 24))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 25))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 26))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 27))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 28))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 29))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 30))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 31))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 33))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim




Code: Select all
newLISP v.10.7.3 64-bit on Windows IPv4/6 libffi, options: newlisp -h

> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 1))
"\000"
> (trim str)
""
> (setq str (dup "\000" 2))
"\000\000"
> (trim str)
""
> (setq str (dup "\000" 3))
"\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 4))
"\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 5))
"\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 6))
"\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 7))
"\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 8))
"\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 9))
"\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 10))
"\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 11))
"\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 12))
"\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 13))
"\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 14))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 15))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 16))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 17))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 18))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 19))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 20))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0"
> (trim str)
""
> (setq str (dup "\000" 21))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000"
> (trim str)
""
> (setq str (dup "\000" 22))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000"
> (trim str)
""
> (setq str (dup "\000" 23))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000"
> (trim str)
""
> (setq str (dup "\000" 24))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 25))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 26))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 27))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 28))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 29))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 30))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 31))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 32))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 33))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 34))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 35))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 36))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 37))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 38))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 39))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
> (trim str)

ERR: not enough memory in function trim
> (setq str (dup "\000" 40))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0"
> (trim str)
""
> (setq str (dup "\000" 41))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000"
> (trim str)
""
> (setq str (dup "\000" 42))
"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000"
> (trim str)
""
(reverse "newlisp")
psilwen
 
Posts: 20
Joined: Thu Jul 03, 2014 5:25 am

Re: Why is the behavior of "trim" function so strange?

Postby psilwen » Fri Aug 18, 2017 6:09 pm

newLISP v.10.7.3
newLISP v.10.7.1
newLISP v.10.7.0
All have this issues.

Usually starting from 8, 16, 24, 32...., depending on the version.

But newLISP v.10.6.2,I tested from 1 to 65536, everything was fine.
(reverse "newlisp")
psilwen
 
Posts: 20
Joined: Thu Jul 03, 2014 5:25 am

Re: Why is the behavior of "trim" function so strange?

Postby HPW » Sat Aug 19, 2017 6:40 am

Hello,

trim seems not to like the null-character.
So is it valid whitespace character?

Code: Select all
newLISP v.10.7.2 32-bit on Windows IPv4/6 libffi, options: newlisp -h
> (setq str "\t\000   \t")
"\t\000   \t"
> (trim str)

ERR: not enough memory in function trim


Code: Select all
> (setq str "\tabc\000   abc\t")
"\tabc\000   abc\t"
> (trim str)
"abc\000   abc"
>


Regards
Hans-Peter
HPW
 
Posts: 1321
Joined: Thu Sep 26, 2002 9:15 am
Location: Germany

Re: Why is the behavior of "trim" function so strange?

Postby Lutz » Sat Aug 19, 2017 4:42 pm

'trim' was overrunning memory when all characters in string < 32 (space).

Fixed here:
http://newlisp.nfshost.com/downloads/de ... nprogress/
Lutz
 
Posts: 5232
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: Why is the behavior of "trim" function so strange?

Postby HPW » Sun Aug 20, 2017 5:41 am

Hello Lutz,

Thanks for the fix.
Will characters <32 now handled as whitespace and get trimmed?
Or do they stay?

Regards
Hans-Peter
HPW
 
Posts: 1321
Joined: Thu Sep 26, 2002 9:15 am
Location: Germany

Re: Why is the behavior of "trim" function so strange?

Postby Lutz » Mon Aug 21, 2017 8:01 pm

See last paragraph here:
http://www.newlisp.org/downloads/develo ... 10.7.3.txt

So the 10.7.3 now is more forgiving when a string contains zero characters, which it should'nt in perfect ASCII or UTF-8 where the zero character marks the end of the string.
Lutz
 
Posts: 5232
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: Why is the behavior of "trim" function so strange?

Postby psilwen » Wed Sep 13, 2017 5:55 am

trim still has bug.

Code: Select all
newLISP v.10.7.3 32-bit on Windows IPv4/6 UTF-8 libffi, options: newlisp -h

> (setq str "\xbb\xe1")
"会"
> (length (trim str))
2
> (length (trim str "\000"))
5
> (setq str "\xce\xaa\xca\xb2\xc3\xb4\xbb\xe1")
"为什么会"
> (length (trim str))
8
> (length (trim str "\000"))
11
(reverse "newlisp")
psilwen
 
Posts: 20
Joined: Thu Jul 03, 2014 5:25 am

Re: Why is the behavior of "trim" function so strange?

Postby TedWalther » Wed Sep 13, 2017 8:20 am

It occurs to me, if I was trimming a string, and it had \000 characters on either the left or the right side, I'd want them GONE. If it is a binary string where the \000 character belongs, then I wouldn't be running trim on it, since trim is a textual function.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 585
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: Why is the behavior of "trim" function so strange?

Postby ralph.ronnquist » Wed Sep 13, 2017 9:24 am

Technically, since neither "\xbb\xe1" nor "\xce\xaa\xca\xb2\xc3\xb4\xbb\xe1" are valid UTF-8 strings, the trim behaviour is conveniently undefined. It appears the trim function expands "\xbb" into a 2-byte code, and "\xe1" into a three byte code, making its output valid UTF-8. ("\xce\xaa", "\xca\xb2" and "\xc3\xb4" are valid UTF-8 characters).
ralph.ronnquist
 
Posts: 171
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia


Return to newLISP in the real world

Who is online

Users browsing this forum: No registered users and 1 guest