Jonathan Sauer wrote:
Well, I experimented a bit and am now a lot wiser. Also, I have confirmation on what I thought all along: My computer's internal harddisk is much too slow:
How to measure the performance of virtual memory:
\directlua0{lua.bytecode[100000000] = function() end}
works ok here (but i have a 4 gig machine); technically it should only allocate one lua state; on the other hand, it may be that a non sparse array is used for the register housekeeping
On a more serious matter, though, there is a buffer overflow in llualib.c, function set_bytecode: If `sizeof(bytecode)*(k+1)' is larger than UINT_MAX, a numeric overflow occurs and not enough memory is allocated in line 162. When initializing the newly allocated bytecode registers afterwards, random memory is overwritten.
Example (without exploit, of course):
Register number = 1000000000:
such overruns happen also when you do wild things with lua code, tex does not manage that memory another overflow can be in the piping data to tex (tex.print) .. if you collect 2 gig data there you may also run into problems
(I killed the process in the latter experiment, as it still consumed too much memory [about 1.9GB]. But: Even though the memory requirements have been higher than in the first example, in this case the allocation did not fail, as the overflow created a smaller allocation, `only' 1935228944 bytes)
Register number = 3000000000:
test \directlua0{lua.bytecode[1024*1024*1024*1024] = function() end} test \directlua0{for i=1,100000000 do lua.bytecode[i] = function() end end } seems to work ok here but test \directlua0{lua.bytecode[3000000000] = function() end} test \directlua0{for i=1,3000000000 do lua.bytecode[i] = function() end end } report a problem with a negavtive value, so it may be that there is a problem there (not sure if taco tests the max value) ! LuaTeX error negative values not allowed. l.7 ...{lua.bytecode[3000000000] = function() end}
IMO it would be sensible to limit the number of bytecode registers to UINT_MAX/sizeof(bytecode) or -- to be platform-independent in the light of 64 bit processors[2] -- (2^32-1)/sizeof(bytecode).
such a limitation is not that meaningful; one can have 1 milion bytecode functions savely but 10 using large datastructures and bombing; there is no control over the lua end of the game; also, when using much data, in practice the garbage collectors will bring down your system (so slow that one will abort the job); luatex kind of assumes modern memory management and machines with memort in the gig range
[3] I think, this is the result of the sig-handler LuaTeX installs which displays an error message. But it seems that this message has been overwritten as well.
error messages and cathing errors with proper messages is on the agenda for next year but segfaults and crashes indeed need to be cached (i can at least imagine a practical limit of 64K bytecode registers)
Of course, all of this is void, if the current bytecode register implementation is only temporary.
not that i know -) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------