"save" makes improper [text] strings; impossible to reload

[TedWalther]

[TedWalther]

As a temporary workaround, before I ever "save" a string, I do (replace {]} my_string {\]}). And then after saving and loading it, I reverse it with (replace {\]} my_string {]})

[Lutz]

The distinction of shorter or bigger-equal than 2048 is done for speed. The internal string representation is the same. What differs is the method used for reading strings from source. For strings shorter than 2048 limited by “” or {}, allocated buffers or the stack and one read() is used. On output newLISP looks at the length than chooses the "" versus [text][/tags]. For longer strings stream reading is used. The short kind, the most used in program source, can also be handled fast on the C stack, while the long cannot.

But the three sets of delimiters also offer the programmer different ways of encoding strings. E.g. for web work [text][/text] are very convenient because they take text as is including line feeds. {} are useful for regular expression for avoiding double backslashing.

[TedWalther]

Lutz, can you change things so that "save" function works? Is save much good if you can't "load" the output?

[Lutz]

see here: http://www.newlisp.org/downloads/develo ... nprogress/

[hartrock]

[TedWalther]

I was thinking, from point of view of correctness, easiest just to have "" for strings. Go language has `` for unescaped strings. {} strings are convenient, but can sometimes bite that they can only contain balanced parenthesis. I think overall, if "" worked for any size, I'd rarely use the other kinds, especially if "" strings allow literal characters inside, so I can use them the same was as [text] and {} for multiline strings.

It is a huge "gotcha" to have a string type (two string types) that work only on some content, but don't have any escape mechanism to include all content. And then the one type of string that does allow all content, is limited to 2048 bytes. Aaarrghghh!

Lutz, how about this; when doing "save", if the string is longer than 2048 bytes, instead of converting it to [text], convert it to (list num num num) where each num is a byte, value in the range 0..255 Or else (string str1 str2...) where str1 is a 2048 byte string in "" representation, as is str2, up until all the content is represented.

[Lutz]

In Hartrock’s example the \000 characters are still in sb but text in [text],[/text] tags does not escape characters, so \000 is not shown, although part of sb.

Right now in 10.6.4 (in progress) base64 encoding is used for strings > 2047 characters and containing the [/text] tag when using save. I could drop the condition for [/text] and always use base64 for strings longer 2047 in the save function. That would make Hartrock’s sb string usable for the save function.

But I don’t want to give up the [text],[/tags] to use completely unescaped text (not binary content). This is frequently used in web programming. I also don’t want to eliminate the 2047 limit for “ quoted string for speed in processing.

The need to display code > 2047 and containing non-displayable binary info is very rare. The save now will work with binary contents too if it always uses base64 transformation on strings > 2047 characters.

[TedWalther]

The base64 solution bothers me, because it breaks the human readability of the (save) output. Can you share a bit more info about the speedups you achieved with the three different string types? How about a 4th type of string, L" for "" strings longer than 2048 bytes?

update

Or instead of L"my string", do it Python style: """my string"""

[hartrock]

[Lutz]

[TedWalther]

[hartrock]

[hartrock]

[Lutz]

After doing a few more benchmarks, I stay with append, which is faster when appending more than a few strings. In the save case you have at least about 30 strings of about 72 characters each. extend does a realloc() on each string, while append allocates memory in bigger chunks.

[hartrock]

[hartrock]

[Lutz]

extend is destructive append is not. Many things in newLISP can be done in either a destructive or non-destructive manner and destructive functions are marked with an ! in the reference.

http://www.newlisp.org/downloads/newlis ... estructive

Most destructive functions - like extend too - can also do in-place modification. See the last sub-chapter about this on this page: http://www.newlisp.org/index.cgi?page=Closures

Bench-marking is a tricky business you can find some benchmarks comparing to Perl and Python here: http://newlisp.nfshost.com/benchmarks/
Small changes how a function is used may cause big differences in benchmarks. Sometimes just moving between OSs or versions of compilers changes everything. That is, why I like the benchmarks in the link, because they always measure function performance in an application context.

ps: people report errors under all topics of the forum, and that has worked well in the passed times.

[hartrock]

[hartrock]

[xytroxon]

[hartrock]

[hartrock]

[Lutz]

Try to boil down the code to the least amount needed to produce the error. Back-tracing does not help here.

[hartrock]