[Dev-luatex] Memory leak in string.explode()?
Hans Hagen
pragma at wxs.nl
Wed Nov 7 10:07:45 CET 2012
On 11/7/2012 1:50 AM, Reinhard Kotucha wrote:
> What I don't grok is why it's 20 times slower to load a file at once
> than in little chunks. But what I'm mostly concerned about is memory
> consumption. In xosview I can see that the garbage collector reduces
> the consumed memory from time to time, but it doesn't seem to be
> overly hungry. Even worse, I have the impression that memory
> consumption grows exponentially with time. With a slightly larger
> test file my system (4GB RAM) would certainly run out of memory.
I think 20 times is somewhat off at your end because here I get this:
all 1.078 34176584 120272.328125
chunked 0.707 34176584 169908.59667969
once 1.129 34176584 111757.03710938
with:
do
collectgarbage("collect")
local m = collectgarbage("count")
local t = os.clock()
local f = io.open("all.xml",'rb')
local d = f:read("*all")
f:close()
print("all",os.clock()-t,#d,collectgarbage("count")-m)
end
do
collectgarbage("collect")
local m = collectgarbage("count")
local d = { }
local t = os.clock()
local f = io.open("all.xml",'rb')
while true do
local r = f:read(2^13)
if not r then
break
else
d[#d+1] = r
end
end
f:close()
d = table.concat(d)
print("chunked",os.clock()-t,#d,collectgarbage("count")-m)
end
do
collectgarbage("collect")
local m = collectgarbage("count")
local t = os.clock()
local f = io.open("all.xml",'rb')
local n = f:seek("end")
f:seek("set",0)
local d = f:read(n)
f:close()
print("once",os.clock()-t,#d,collectgarbage("count")-m)
end
When doing such tests, make sure that you do a garbage collection run
and clean up old variables.
If I remember right, luatex has a patched buffersize (and fast loading
when using "all") because we did similar tests in the beginning. What I
don't understand is that the *all is so much slower. In fact, it should
be faster because only one string has to be hashed, unless deep down,
lua collects small snippets and hashes them.
Actually, I always load with "all" because in the beginning it was way
faster, so something is messed up. The chuncked approach uses more memory.
Hans
-----------------------------------------------------------------
Hans Hagen | PRAGMA ADE
Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
| www.pragma-pod.nl
-----------------------------------------------------------------
More information about the dev-luatex
mailing list