[Dev-luatex] Memory leak in string.explode()?

[Hans Hagen]

On 11/7/2012 1:50 AM, Reinhard Kotucha wrote:

> What I don't grok is why it's 20 times slower to load a file at once
> than in little chunks.  But what I'm mostly concerned about is memory
> consumption.  In xosview I can see that the garbage collector reduces
> the consumed memory from time to time, but it doesn't seem to be
> overly hungry.  Even worse, I have the impression that memory
> consumption grows exponentially with time.  With a slightly larger
> test file my system (4GB RAM) would certainly run out of memory.

I think 20 times is somewhat off at your end because here I get this:

all	1.078	34176584	120272.328125
chunked	0.707	34176584	169908.59667969
once	1.129	34176584	111757.03710938

with:

do
     collectgarbage("collect")
     local m = collectgarbage("count")
     local t = os.clock()
     local f = io.open("all.xml",'rb')
     local d = f:read("*all")
     f:close()
     print("all",os.clock()-t,#d,collectgarbage("count")-m)
end

do
     collectgarbage("collect")
     local m = collectgarbage("count")
     local d = { }
     local t = os.clock()
     local f = io.open("all.xml",'rb')
     while true do
         local r = f:read(2^13)
         if not r then
             break
         else
             d[#d+1] = r
         end
     end
     f:close()
     d = table.concat(d)
     print("chunked",os.clock()-t,#d,collectgarbage("count")-m)
end

do
     collectgarbage("collect")
     local m = collectgarbage("count")
     local t = os.clock()
     local f = io.open("all.xml",'rb')
     local n = f:seek("end")
     f:seek("set",0)
     local d = f:read(n)
     f:close()
     print("once",os.clock()-t,#d,collectgarbage("count")-m)
end

When doing such tests, make sure that you do a garbage collection run 
and clean up old variables.

If I remember right, luatex has a patched buffersize (and fast loading 
when using "all") because we did similar tests in the beginning. What I 
don't understand is that the *all is so much slower. In fact, it should 
be faster because only one string has to be hashed, unless deep down, 
lua collects small snippets and hashes them.

Actually, I always load with "all" because in the beginning it was way 
faster, so something is messed up. The chuncked approach uses more memory.

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------