[Dev-luatex] Dump sharing - redundant copying/allocations

Hans Hagen j.hagen at xs4all.nl
Wed Jul 21 09:09:29 CEST 2021

On 7/20/2021 9:51 PM, Michal Vlasák wrote:
> On Tue Jul 20, 2021 at 7:56 PM CEST, Taco Hoekwater wrote:
>> Hi,
>>> On 20 Jul 2021, at 18:01, Michal Vlasák <lahcim8 at gmail.com> wrote:
>>> Hello,
>>> when NO_DUMP_SHARE is not defined and the system is Little endian, then
>>> for the possibility of sharing format files between architectures all
>>> dumped multi-byte valules are byte swapped by the function "swap_items"
>>> in "tex/texfileio.c".
>>> The origin of the function is in Web2C's "texmfmp.c" and it seems that
>>> it was added to LuaTeX in the version 1.08 when the source was converted
>>> from CWeb (or am I reading the history wrong?).
>> That code has been around since 2010 (luatex 0.60-ish). I can’t
>> remember exactly why I did that odd copying to a temp array, but I do
>> remember that that particular function was quite problematic w.r.t.
>> endianness and (cross)compiler issues (read: bugs, as in “internal
>> compiler error").
> Thank you for your valuable insight Taco. My thinking was skewed because
> I read the history incorrectly, thinking it was more of a mistake that
> slipped into a big commit.
> Anyways despite the number of allocations the performance is probably
> fine, since nobody complained, yet. I noticed it by chance while reading
> the code.
just afew remarks:

- dump sharing in luatex makes no sense, also because lua byte code can 
be stored and that is not portable .. for that reason byte swapping was 
removed at some later point in the project

- byte swapping introduces overhead and was happening for the majority 
of users (intel), but the code was/is still there

- we store a format version number in the format so that a format file 
will not be loaded when there is a mismatch (that was added later)

- format gz compression was introduced already early to keep the format 
small and irr that needed some of these copying tweaks

- as you mention, the allocation overhead is small, which is definitely 
true compared to byte swapping and all the decompression calls in 
between (loading the file database at startup probably takes more time, 
and definitely in the early days was quite noticeable, more than format 
loading); the level 3 compression gav ethe best trade-off

- the mentioned 'internal compiler error' mentioned by Taco rings a 
bell, it's a reason why often saving/loading an 'int' goes via a 
variable because compilers would optimize in a way that dumping 
variables (ints) in more complex data structures gave issues

- there are a few more places where using redundant temp vars are used 
because during some operations memory can grow which can makes pointers 
already set invalid, some of those have been sorted out differently in 
the meantime)

- talking of performance, one of the interesting things in the beginning 
of development was that we noticed different (incremental) versions to 
perform differently; for instance when math was opened up the machinery 
became real slow, as if we crossed some boundary, (compilation order of 
specific code modules mattered too); but when i then updated my laptop 
it was fast again, not so much because of fhe faster cpu but because the 
cpu cache was larger; compiler optimization also kind of interfered (at 
that time ideas, experiments and binaries came and went on a daily 
basis, we had quite some fun)

- if performance is of concern, we also noticed (later one when luajit 
enteres the scenary) that the settings for lua hashing matters, and that 
luajit had pretty bad heuristics (tuned for url, we published about 
that) so we used a different hashing there ...


                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl

More information about the dev-luatex mailing list