On Wed Jul 21, 2021 at 9:09 AM CEST, Hans Hagen wrote:
just afew remarks:
- dump sharing in luatex makes no sense, also because lua byte code can be stored and that is not portable .. for that reason byte swapping was removed at some later point in the project
Funnily enough, I looked at the code exactly because of this. As it turns out, in TeX Live after recent problems with format sharing across 32-bit Windows and 64-bit Linux, there is now an effort to ensure the portability of formats: https://git.texlive.info/texlive/tree/Master/tlpkg/bin/tl-check-fmtshare Of course the issue of unportable bytecode came up. As far as I know three LuaTeX formats store it in format files: ConTeXt (wasn't and probably won't be checked by the script and users alike), OpTeX and minim (recent format, not even genereted in TeX Live). I evaluated the possibility of byte swapping in the Lua (un)dumping, by introducing a patch in TeX Live, but I don't think it is worth: - I personally wouldn't encourage _more_ format sharing between OS's and architectures in the future. - The types used by Lua (long long, double, int) may not even be portable anyways. - The "right" approach for portable dumping doesn't fit the current architecture. (https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html)
- byte swapping introduces overhead and was happening for the majority of users (intel), but the code was/is still there
I agree that if something the more common Little endian would have been a better choice.
- as you mention, the allocation overhead is small, which is definitely true compared to byte swapping and all the decompression calls in between (loading the file database at startup probably takes more time, and definitely in the early days was quite noticeable, more than format loading); the level 3 compression gav ethe best trade-off
Interestingly I didn't find any really measurable slowdown from the byte swapping, possibly because the entire function was turned into a lot of SIMD instructions. (But to be fair I didn't test with a huge format, and usually compile LuaTeX with the byte swapping disabled anyways.)
- the mentioned 'internal compiler error' mentioned by Taco rings a bell, it's a reason why often saving/loading an 'int' goes via a variable because compilers would optimize in a way that dumping variables (ints) in more complex data structures gave issues
- there are a few more places where using redundant temp vars are used because during some operations memory can grow which can makes pointers already set invalid, some of those have been sorted out differently in the meantime)
Very interesting, hopefully the situation improved since.
- talking of performance, one of the interesting things in the beginning of development was that we noticed different (incremental) versions to perform differently; for instance when math was opened up the machinery became real slow, as if we crossed some boundary, (compilation order of specific code modules mattered too); but when i then updated my laptop it was fast again, not so much because of fhe faster cpu but because the cpu cache was larger; compiler optimization also kind of interfered (at that time ideas, experiments and binaries came and went on a daily basis, we had quite some fun)
- if performance is of concern, we also noticed (later one when luajit enteres the scenary) that the settings for lua hashing matters, and that luajit had pretty bad heuristics (tuned for url, we published about that) so we used a different hashing there ...
Thank you for your insights and caring about performance! Michal Vlasák