On Wed, Nov 7, 2012 at 10:07 AM, Hans Hagen <pragma@wxs.nl> wrote:
On 11/7/2012 1:50 AM, Reinhard Kotucha wrote:

What I don't grok is why it's 20 times slower to load a file at once
than in little chunks.  But what I'm mostly concerned about is memory
consumption.  In xosview I can see that the garbage collector reduces
the consumed memory from time to time, but it doesn't seem to be
overly hungry.  Even worse, I have the impression that memory
consumption grows exponentially with time.  With a slightly larger
test file my system (4GB RAM) would certainly run out of memory.

I think 20 times is somewhat off at your end because here I get this:
Out of memory here  with a testfile of  662M
Linux 32bit, 4GByte, PAE extension

# time ./read_blocks.lua

real    0m2.458s
user    0m0.372s
sys    0m1.084s
# time ./read_whole_file.lua
not enough memory

real    0m17.125s
user    0m11.737s
sys    0m4.292s

# texlua -v
This is LuaTeX, Version beta-0.70.1-2012052416 (rev 4277)

(i.e. not the patched one)
--
luigi