Out of memory message

[John_Small]

I'm using newLISP for filtering large number of
huge ascii text files (one line equals one record).

The filter is making simple changes to the input
text line (record) and printing it to a tmp file
and then when finished replacing the original
file (after it was closed of course).

After running for an hour or so it finally
crashes with a message to the effect it is
out of memory in function such and such.

I'm not using any recursive algorithm so
I don't think it is the stack running out of
space. I am running a huge number of
huge files through the filter (using "for"
loops to read the archive directory files).
The filter is making changes to these
archive files and this is why it can run
for hours.

I didn't save the exact error message but
I can start the job again and get the message
verbatim if you need it.

[cormullion]

i'd be happy to run something similar on my machine to see if it does the same. Need some similar code, though...!

[John_Small]

Do you have an email that I can send you the
code?

It is basically a generic filter that anyone can use.
The configuration file customizes it for handling
your particular flat files.

You simply specify the match column regular expressions
and the replace column RE's and replacement text.

So this could be a useful utility if I can eliminate the out
of memory crash.

[rickyboy]

John, Send it to me too -- you know my email. --Ricky

[cormullion]

Yes, I think there is something going on. After about 10 minutes Virtual Memory usage has gone from 50MB to 300MB... Increasing by about 10MB every two seconds. If you ctrl-c and enter the debugger while this is running you get:

(sys-info)
(12191358 268435456 383 16 28 2048 9100 131)

and then a minute later:

(sys-info)
(12653445 268435456 383 16 28 2048 9100 131)

I'm no boffin, but I can see the first number increasing inexorably. After some time presumably this would exceed some limit?

How, though, do you know where all these extra bytes are going, though? Ah, there I need some advice...

[John_Small]

I'm not sure I understand what you are
asking.

Unless I'm unwittingly recursively calling
something and growing the stack what else
could be causing this memory leak? Is
it my code or the newLISP garbage collector?

TBTW - hanks for taking the time to run it.

[cormullion]

I think your line-xform is 'growing the stack' (no idea what that means but it sounds good). The number of cells is larger at the end of the function than at the start...

Edit: No, actually my previous post shows that it's not the stack that's growing. It's the number of cells that's growing inexorably...

[rickyboy]

What data should I use to test this with? Made-up data or yours? Yours would be better, but if I need to make up any data, please give me some specs that you think relevant. Thanks! --Rick

[rickyboy]

John,

In the for loop, in routine line-xform, you use n as the index; then inside that loop you re-assign n (in numerous places) which is screwing you to high hell. Use lets not setqs, and you will be fine -- probably not blow out memory anymore.

--Ricky

[John_Small]

I forgot about dynamic scoping.

I wrote this when late at night when
I was half asleep.

Thanks!

[cormullion]

glad you identified the problem..

It's cool stuff. Do you write much about newLISP these days? (I'm assuming you're John "newLISP in 21 minutes" Small I suppose...)

cheers

[rickyboy]

Funny you should ask that cormullion. I also asked that about a month ago here.

The answer is YES, he is the John "newLISP in 21 minutes" Small. :-)

[cormullion]

Yes, I thought he might be - and he has told me he is as well!

I've just been trying to get him to write more stuff for us - 21 minutes is OK, but I'm ready for another half hour now! :-)

[rickyboy]

I've just been trying to get him to write more stuff for us - 21 minutes is OK, but I'm ready for another half hour now! :-)

I agree. John writes very well (as you do also cormullion), so another half-hour would be welcome.

[John_Small]

I think I should warm up on newLISP again
especially on contexts and def-new and the
new unify etc.

I'll try to then write about these additional subjects

1. lexically scoped programming in newLISP
codified with examples (and perhaps macros)
to make it as easily as rolling off a log.

2. a closer look at macros (newLISP macros
have real potential for exploitation I think
and needs very carefully study to see exactly
how potent it can be.)

3. explore the unify and see if we can fashion
continuations through CPS (continuation passing style)
perhaps using macros in order to facilitate non deterministic
programming. This is where we have to have
lexical scoping or it will be hopeless. If we can do
something useful in the way of non determinism (easily)
then newLISP starts to become compelling. Otherwise
unify is just a nice way to get around "let" and maybe
cracking and mapping structures (which is worthy discussing
anyway). If we can get non determinism working than
folding this back onto macros can become extremely
interesting as we build non deterministic JIT macros.

I also want to rewrite the flatreplace.lsp or read
what Rick Hanson sent me as soon as I can get
a free minute which might be 2 weeks from now.
I want to test this memory problem again. Even
though my code was buggy the memory problem
should not have been a problem since I was not
recursively calling anything. I suspect the problem
is in Windows memory allocation since a port to
PHP of flatreplace also died several hours into
the batch job.