[Dev-luatex] Memory leak in string.explode()?
Taco Hoekwater
taco at elvenkind.com
Wed Nov 7 11:21:15 CET 2012
On 11/07/2012 11:07 AM, Hans Hagen wrote:
> On 11/7/2012 10:33 AM, luigi scarso wrote:
>>
>>
>> On Wed, Nov 7, 2012 at 10:07 AM, Hans Hagen <pragma at wxs.nl
>> <mailto:pragma at wxs.nl>> wrote:
>>
>> On 11/7/2012 1:50 AM, Reinhard Kotucha wrote:
>>
>> What I don't grok is why it's 20 times slower to load a file
>> at once
>> than in little chunks. But what I'm mostly concerned about is
>> memory
>> consumption. In xosview I can see that the garbage collector
>> reduces
>> the consumed memory from time to time, but it doesn't seem to be
>> overly hungry. Even worse, I have the impression that memory
>> consumption grows exponentially with time. With a slightly
>> larger
>> test file my system (4GB RAM) would certainly run out of memory.
>>
>>
>> I think 20 times is somewhat off at your end because here I get this:
>>
>> Out of memory here with a testfile of 662M
>> Linux 32bit, 4GByte, PAE extension
>>
>> # time ./read_blocks.lua
>>
>> real 0m2.458s
>> user 0m0.372s
>> sys 0m1.084s
>> # time ./read_whole_file.lua
>> not enough memory
>>
>> real 0m17.125s
>> user 0m11.737s
>> sys 0m4.292s
>>
>> # texlua -v
>> This is LuaTeX, Version beta-0.70.1-2012052416 (rev 4277)
>
> Indeed not enough mem on my laptop for a 600M+ test.
>
> Windows 8, 32 bit:
>
> -- all 1.082 34176584 120272.328125
> -- chunked 0.668 34176584 169908.59667969
> -- once 1.065 34176584 111757.03710938
>
> -- all 7.078 136706337 535063.34863281
> -- chunked 3.441 136706337 787195.56933594
> -- once 6.621 136706337 501559.83691406
>
> the larger values for 'all' and 'once' still puzzle me.
malloc time, perhaps. The 'fast' *a loader does a couple of fseek()
ftell()s to find the file size, then malloc()s the whole string
before feeding it to Lua, then free-ing it again. There is a
fairly large copy in the feed process that I cannot avoid without
using lua internals instead of the published API.
Btw, on my SSD disk, there is no noticeable difference between all
three cases for an 85MB file.
More information about the dev-luatex
mailing list