de-tag

[newdep]

Hello All,

Anyone tried to build a html de-tagger from string/buffer?

Norman.

[Lutz]

Try this:

Code: Select all

(define (strip-html buff)
    (set 'page '())
    (dolist (lne (parse buff "\r\n|\n" 0))
        (push (replace "<.+?>" lne " " 0) page))
    (join (reverse page) "\n"))

(strip-html (get-url "http://newlisp.org"))

On smaller buffers (replace "<.+?>" buff " " 0) may be just enough code, but on bigger numbers the line-by-line solution will be much faster.

You may also want to try the 'greedy' option 512 instead of 0 for better/faster results.

Lutz

[newdep]

Thank you lutz, you also answered a small question on a previous post
with the strip-html example.. I always forget to move towards list from string..
I should think more into lists using newlisp :-)

Norman.

[Lutz]

I did some benchmarks (strip-html buff) versus (replace "<.+?>" buff " " 512) and it turns out that the simpler solution without splitting into lines is also the fastest on 'newlisp_manual.html' a 400Kbyte file.

The biggest speedup is using the greedy option 512, cutting the time to a quarter:

(strip-html buff) -> 12 seconds

(replace "<.+?>" buff " " 0) 13 seconds

(replace "<.+?>" buff " " 512) 3 seconds !!!

Lutz

ps: still don't forget the push/join method, which sometimes is superior (see base64 example). Also: 'greedy' gives different output!

[William James]

Code: Select all

> (regex {<.*>} "Be <b>very</b> careful.")
("<b>very</b>" 3 11)
> (regex {<.*?>} "Be <b>very</b> careful.")
("<b>" 3 3)
> (regex {<.*>} "Be <b>very</b> careful." 512)
("<b>" 3 3)
> (regex {<.*?>} "Be <b>very</b> careful." 512)
("<b>very</b>" 3 11)

In .*?, ? makes the * non-greedy. Since option 512 inverts greediness, (regex {<.*?>} "..." 512) is equivalent to (regex {<.*>} "..."). Both of these may strip out too much from the html string. However, note that the dot, as in Ruby and Perl, won't match a newline unless you select that option.

Code: Select all

> (setq str "<foo\nbar>")
"<foo\nbar>"
> (replace {<.*>} str "" 0)
"<foo\nbar>"
> str
"<foo\nbar>"
> (replace {<.*>} str "" 4)
""
> str
""