Hi all, I have a rather large (almost 120,000 lines) xml file, processing pulls in another large xml file. The project compiles under mkiv; with lmtx, I get tex error > tex error on line 8 in file : TeX capacity exceeded, sorry [reference count=32767] <macro> \reallanguagetag just twenty pages shy of the end of the document. There is an error log with this content: return { ["errortype"]=3, ["filename"]="", ["lastcontext"]="\n\n\n<macro> \\reallanguagetag \n \n<macro> \\reallanguagetag #1->\\ifcsname \\??languagelinked #1\n \\endcsname \\lastnamedcs \\else #1\\fi \n<macro> \\setusedlanguage #1->\\edef \\currentusedlanguage {\\reallanguagetag {#1}\n }\\ifempty \\currentusedlanguage \\let \\currentusedlanguage \\currentlanguage \\always\\let \\usedlanguageparameter \\languageparameter \\orelse \\ifx \\currentusedlanguage \\v!global \\let \\currentusedlanguage\n\n<macro> \\typo_delimited_push ...\\currentdelimitedtext {#1}\\edef \\m_delimited_argument {#2}\\ifempty \\m_delimited_argument \\typo_delimited_set_language_nop \n \\else \\typo_delimited_set_language_yes \\fi \\let \\currentparentdelimitedtext \\currentdelimitedtext \\global \\advance \\c_typo...\n\n<macro> \\delimitedtext [#1]#*[#2]->\\dontleavehmode \\typo_delimited_push {#1}{#2}\n \\edef \\p_delimited_method {\\delimitedtextparameter \\c!method }\\ifx \\p_delimited_method \\v!font \\expandafter \\typo_delimited_fontdriven \\else \\expandafter \\typo_delimited_other \\fi \n\n<to be read again> \n {\n...\n\n<initial> \n ", ["lastluaerror"]="", ["lasttexerror"]="TeX capacity exceeded, sorry [reference count=32767]", ["lasttexhelp"]="If you really absolutely need more capacity, you can ask a wizard to enlarge me.", ["linenumber"]=8, ["offset"]=10, ["skiplinenumber"]=8, } In mkiv, these are the stats at the end of the run: mkiv lua stats > used config file: selfautoparent:/texmf/web2c/texmfcnf.lua mkiv lua stats > used cache path: /mnt/shared/lmtx/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e mkiv lua stats > resource resolver: loadtime 0.009 seconds, 1 scans with scantime 0.005 seconds, 0 shared scans, 27 found files, scanned paths: /home/tas/texmf mkiv lua stats > stored bytecode data: 455 modules (0.122 sec), 96 tables (0.007 sec), 551 chunks (0.130 sec) mkiv lua stats > traced context: maxstack: 1422, freed: 2, unreachable: 1420 mkiv lua stats > cleaned up reserved nodes: 63 nodes, 9 lists of 442 mkiv lua stats > node memory usage: 17 attribute, 4 attribute_list, 1 glue, 51 glue_spec, 3 kern, 2 penalty, 2 temp, 2 user_defined mkiv lua stats > node list callback tasks: 8 unique task lists, 9336 instances (re)created, 177413 calls mkiv lua stats > h-node processing time: 4.800 seconds including kernel mkiv lua stats > attribute processing time: 0.617 seconds front- and backend mkiv lua stats > driver time: 0.886 seconds mkiv lua stats > used backend: pdf (backend for directly generating pdf output) mkiv lua stats > jobdata time: 0.161 seconds saving, 0.052 seconds loading mkiv lua stats > callbacks: internal: 261417, file: 319, direct: 3, late: 31831, function 200128, total: 493698 (464 per page) mkiv lua stats > randomizer: resumed with value 0.47143577262518 mkiv lua stats > positions: 9535 collected, 9535 regular, 0 special mkiv lua stats > v-node processing time: 0.401 seconds mkiv lua stats > result saved in file: plutarchusvitae.pdf, compresslevel 3, objectcompresslevel 1 mkiv lua stats > loaded patterns: de::3 en::1 la::65, load time: 0.000 mkiv lua stats > xml load time: 0.648 seconds, 4 files, 4 converted mkiv lua stats > lxml preparation time: 0.626 seconds, 233814 nodes, 578788 lpath calls, 578743 cached calls mkiv lua stats > interactive elements: 0 references, 37 destinations mkiv lua stats > margin data: 10798 entries, 0 pending mkiv lua stats > loaded fonts: 9 files: garamondpremrpro.otf, sabonltstd-bold.otf, sabonltstd-italic.otf, sabonltstd-roman.otf, kadmosu.otf, oldstandard-bold.otf, oldstandard-italic.otf, xitsmath-regular.otf, dejavusansmono.ttf mkiv lua stats > font engine: otf 3.113, afm 1.513, tfm 1.000, 17 instances, 6 shared in backend, 0 common vectors, 6 common hashes, load time 0.146 seconds mkiv lua stats > used platform: linux-64, type: unix, binary subtree: texmf-linux-64 mkiv lua stats > used engine: luatex version: 1.13, functionality level: 7420, banner: this is luatex, version 1.13.0 (tex live 2021) mkiv lua stats > used hash slots: 47131 of 65536 + 250000 mkiv lua stats > lua properties: engine: lua 5.3, used memory: 601 MB, ctx: 579 MB, max: unknown MB, hash chars: min(64,40), symbol mask: utf (τεχ) mkiv lua stats > runtime: 22.310 seconds, 1068 processed pages, 1068 shipped pages, 47.871 pages/second system | total runtime: 213.928 seconds Is texmfcnf.lua still the configuration file I need to look at for this kind of problem? Which variable could I change? Thanks, and all best Thomas
On 5/17/2021 11:43 PM, Thomas A. Schmitz wrote:
Hi all,
I have a rather large (almost 120,000 lines) xml file, processing pulls in another large xml file. The project compiles under mkiv; with lmtx, I get
tex error > tex error on line 8 in file : TeX capacity exceeded, sorry [reference count=32767]
This is not the normal memory consumption that bothers you \enabletrackers[system.usage] will show you more detail. Normally the default context configuration will provide enough. In lmts memory is abit more dynamic so we start otu smaller and then stepwise grow. And one can configure even more (actually i just updated some configuration options for that). Anyway: this more smells like some nesting going wrong and therefore excessive token sharing
<macro> \reallanguagetag
just twenty pages shy of the end of the document.
There is an error log with this content:
return { ["errortype"]=3, ["filename"]="", ["lastcontext"]="\n\n\n<macro> \\reallanguagetag \n \n<macro> \\reallanguagetag #1->\\ifcsname \\??languagelinked #1\n \\endcsname \\lastnamedcs \\else #1\\fi \n<macro> \\setusedlanguage #1->\\edef \\currentusedlanguage {\\reallanguagetag {#1}\n }\\ifempty \\currentusedlanguage \\let \\currentusedlanguage \\currentlanguage \\always\\let \\usedlanguageparameter \\languageparameter \\orelse \\ifx \\currentusedlanguage \\v!global \\let \\currentusedlanguage\n\n<macro> \\typo_delimited_push ...\\currentdelimitedtext {#1}\\edef \\m_delimited_argument {#2}\\ifempty \\m_delimited_argument \\typo_delimited_set_language_nop \n \\else \\typo_delimited_set_language_yes \\fi \\let \\currentparentdelimitedtext \\currentdelimitedtext \\global \\advance \\c_typo...\n\n<macro> \\delimitedtext [#1]#*[#2]->\\dontleavehmode \\typo_delimited_push {#1}{#2}\n \\edef \\p_delimited_method {\\delimitedtextparameter \\c!method }\\ifx \\p_delimited_method \\v!font \\expandafter \\typo_delimited_fontdriven \\else \\expandafter \\typo_delimited_other \\fi \n\n<to be read again> \n {\n...\n\n<initial> \n ", ["lastluaerror"]="", ["lasttexerror"]="TeX capacity exceeded, sorry [reference count=32767]", ["lasttexhelp"]="If you really absolutely need more capacity, you can ask a wizard to enlarge me.", ["linenumber"]=8, ["offset"]=10, ["skiplinenumber"]=8, }
In mkiv, these are the stats at the end of the run:
mwe ...
mkiv lua stats > used config file: selfautoparent:/texmf/web2c/texmfcnf.lua mkiv lua stats > used cache path: /mnt/shared/lmtx/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e
mkiv lua stats > resource resolver: loadtime 0.009 seconds, 1 scans with scantime 0.005 seconds, 0 shared scans, 27 found files, scanned paths: /home/tas/texmf mkiv lua stats > stored bytecode data: 455 modules (0.122 sec), 96 tables (0.007 sec), 551 chunks (0.130 sec) mkiv lua stats > traced context: maxstack: 1422, freed: 2, unreachable: 1420 mkiv lua stats > cleaned up reserved nodes: 63 nodes, 9 lists of 442 mkiv lua stats > node memory usage: 17 attribute, 4 attribute_list, 1 glue, 51 glue_spec, 3 kern, 2 penalty, 2 temp, 2 user_defined mkiv lua stats > node list callback tasks: 8 unique task lists, 9336 instances (re)created, 177413 calls mkiv lua stats > h-node processing time: 4.800 seconds including kernel mkiv lua stats > attribute processing time: 0.617 seconds front- and backend mkiv lua stats > driver time: 0.886 seconds mkiv lua stats > used backend: pdf (backend for directly generating pdf output) mkiv lua stats > jobdata time: 0.161 seconds saving, 0.052 seconds loading mkiv lua stats > callbacks: internal: 261417, file: 319, direct: 3, late: 31831, function 200128, total: 493698 (464 per page) mkiv lua stats > randomizer: resumed with value 0.47143577262518 mkiv lua stats > positions: 9535 collected, 9535 regular, 0 special mkiv lua stats > v-node processing time: 0.401 seconds mkiv lua stats > result saved in file: plutarchusvitae.pdf, compresslevel 3, objectcompresslevel 1 mkiv lua stats > loaded patterns: de::3 en::1 la::65, load time: 0.000 mkiv lua stats > xml load time: 0.648 seconds, 4 files, 4 converted mkiv lua stats > lxml preparation time: 0.626 seconds, 233814 nodes, 578788 lpath calls, 578743 cached calls mkiv lua stats > interactive elements: 0 references, 37 destinations mkiv lua stats > margin data: 10798 entries, 0 pending mkiv lua stats > loaded fonts: 9 files: garamondpremrpro.otf, sabonltstd-bold.otf, sabonltstd-italic.otf, sabonltstd-roman.otf, kadmosu.otf, oldstandard-bold.otf, oldstandard-italic.otf, xitsmath-regular.otf, dejavusansmono.ttf mkiv lua stats > font engine: otf 3.113, afm 1.513, tfm 1.000, 17 instances, 6 shared in backend, 0 common vectors, 6 common hashes, load time 0.146 seconds mkiv lua stats > used platform: linux-64, type: unix, binary subtree: texmf-linux-64 mkiv lua stats > used engine: luatex version: 1.13, functionality level: 7420, banner: this is luatex, version 1.13.0 (tex live 2021) mkiv lua stats > used hash slots: 47131 of 65536 + 250000 mkiv lua stats > lua properties: engine: lua 5.3, used memory: 601 MB, ctx: 579 MB, max: unknown MB, hash chars: min(64,40), symbol mask: utf (τεχ) mkiv lua stats > runtime: 22.310 seconds, 1068 processed pages, 1068 shipped pages, 47.871 pages/second system | total runtime: 213.928 seconds
hardly demanding in terms of memory Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 5/17/2021 11:43 PM, Thomas A. Schmitz wrote:
Is texmfcnf.lua still the configuration file I need to look at for this kind of problem? Which variable could I change? Normally not needed but this is the section for luatex:
["luatex.errorline"] = 250, -- no need to change ["luatex.halferrorline"] = 125, -- no need to change ["luatex.expanddepth"] = 10000, -- no need to change ["luatex.hashextra"] = 100000, -- rarely more is needed ["luatex.nestsize"] = 1000, -- often an error if more needed ["luatex.maxinopen"] = 500, -- i use 2000 ["luatex.maxprintline"] = 10000, -- no need to change ["luatex.maxstrings"] = 500000, -- maybe sometimes ["luatex.paramsize"] = 25000, -- seldom ["luatex.savesize"] = 100000, -- often an error if more needed ["luatex.stacksize"] = 100000, -- often an error if more needed not, in luametatex we have some more and also more control; this is what the next version will support (the above method is how it currently goes, max values that use the preset allocations steps) ["luametatex.errorlinesize"] = { size = 250 }, -- max = 255 ["luametatex.halferrorlinesize"] = { size = 250 }, -- max = 255 ["luametatex.expandsize"] = { size = 10000 }, -- max = 1000000 ["luametatex.stringsize"] = { size = 500000, step = 100000 }, -- max = 2097151 -- number of strings ["luametatex.poolsize"] = { size = 10000000, step = 1000000 }, -- max = 100000000 -- chars in string ["luametatex.hashsize"] = { size = 250000, step = 100000 }, -- max = 2097151 ["luametatex.nodesize"] = { size = 50000000, step = 500000 }, -- max = 50000000 ["luametatex.tokensize"] = { size = 10000000, step = 250000 }, -- max = 10000000 ["luametatex.buffersize"] = { size = 10000000, step = 1000000 }, -- max = 100000000 ["luametatex.inputsize"] = { size = 100000, step = 10000 }, -- max = 100000 -- aka stack ["luametatex.filesize"] = { size = 2000, step = 200 }, -- max = 2000 ["luametatex.nestsize"] = { size = 10000, step = 1000 }, -- max = 10000 ["luametatex.parametersize"] = { size = 100000, step = 10000 }, -- max = 100000 ["luametatex.savesize"] = { size = 500000, step = 10000 }, -- max = 500000 ["luametatex.fontsize"] = { size = 250, step = 250 }, -- max = 100000 ["luametatex.languagesize"] = { size = 250, step = 250 }, -- max = 10000 ["luametatex.marksize"] = { size = 50, step = 50 }, -- max = 10000 One can just define the max if needed because we bump memory stepwise (on demand). It depends on the kind of projects you have. One can also sey { plus = 10000 } to add more to the set size without bothering about existing values. Now, if you watch closely you'll see that most max values are high already (initial values cannot be set) but token and node memory as well as string related values can be set much larger. However, it makes sense to limit. Actually for some scenarios one might even want to go lower. When you compare mkiv and mkxl you'll notice that we use less memory anyway (for various reasons / optimizations). The format file is also smaller (esp when you take into account that the mkiv one is zip compressed and the lmtx one isn't). But ... your problem was not related to the above. The next upload will also mention detailed mem usage in the log file. If you want to see how your document behaves you can say \enabletrackers[system.usage] at the top (is already there for a while). Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 5/17/2021 11:43 PM, Thomas A. Schmitz wrote:
Hi all,
I have a rather large (almost 120,000 lines) xml file, processing pulls in another large xml file. The project compiles under mkiv; with lmtx, I get
tex error > tex error on line 8 in file : TeX capacity exceeded, sorry [reference count=32767]
Can you check the latest upload? You should be able to go way higher. If you're worried about your lua macros performance you can always run with: --profile and look at the luatex-profile.log file ... it gives a rough impression. be prepared for a slow run (although the latest greatest lmtx should perform some 2.5 times better when profiling). There's also a new (yet unofficial) feature: \starttext \setupfonts [serif=dejavuserif*default, sans=dejavusans, mono=dejavusansmono] { AAA\char1234 AAA} {\ss BBB\char1234 BBB}\par { AAAAAA} {\ss BBBBBB}\par \dorecurse{1000}{ \char1234 } \stoptext You can set up a list of 'quick and dirty fallback' fonts (you can set a sequence for each style). This doesn't replace the normal fallback mechanisms but can be handy as catch. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On 19.05.21 18:34, Hans Hagen wrote:
Can you check the latest upload? You should be able to go way higher.
Yes, with 2021.05.19 18:22, the file compiles again in lmtx! I'm also trying your suggestions for improving the Lua code and rethinking my tables for analyzing the vocabulary. Thank you for the quick fix! And I hope there won't be any problem for other users - I'm aware that memory on computers can differ vastly. I sometimes run ConTeXt on my raspberry pi 3, which has just 1 G of RAM, whereas my newish iMac at home has 40 G. So it's difficult to find the sweet spot that will satisfy everybody... All best Thomas
On 5/19/2021 6:49 PM, Thomas A. Schmitz wrote:
On 19.05.21 18:34, Hans Hagen wrote:
Can you check the latest upload? You should be able to go way higher.
Yes, with 2021.05.19 18:22, the file compiles again in lmtx! I'm also trying your suggestions for improving the Lua code and rethinking my tables for analyzing the vocabulary. Thank you for the quick fix! And I hope there won't be any problem for other users - I'm aware that memory
If they have problems it's because of other changes -)
on computers can differ vastly. I sometimes run ConTeXt on my raspberry pi 3, which has just 1 G of RAM, whereas my newish iMac at home has 40 G. So it's difficult to find the sweet spot that will satisfy everybody...
Unless you use many huge fonts, running your document on a rpi 4 with 4-8 gb should work ok (we have a rpi 4 in mojcas compile farm connected to an usb3 ssd). My laptop is from 2013 (decent intel 3840QM cpu + 16 GB mem and ssd and so) and is about 3 times slower than the rpi 4 in the luametatex manual. Your file in mkiv does a decent ppm (is that an arm or intel mac?) mkiv lua stats > lua properties: engine: lua 5.3, used memory: 601 MB, ctx: 579 MB, max: unknown MB, hash chars: min(64,40), symbol mask: utf (τεχ) mkiv lua stats > runtime: 22.310 seconds, 1068 processed pages, 1068 shipped pages, 47.871 pages/second so I suppose in lmtx it's a bit faster and using less mem. Still less mem than the GB's that e.g. firefox consumes after a few hours sitting idle. Concerning the sweetspot: I supose that on a laptop or desktop users are okay, but in a vm (or instance) it might matter because there processes have to fight over memory and cache (some is gained back by often faster disk io because that gets delegated to another process). I try to keep that in mind with luametatex/lmtx, which is why I spent quite some time on a lower mem footprint. I have no clue what a modern cpu does compared to mine (tex is single core). (btw, the latest version should also be a bit more efficient / faster with some huge cjk fonts that for some reasons have so many zero entries that the tma file was 3 times larger than needed, but I suppose fonts will always be a bit of an issue ... they have always been since tex showed up.) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
participants (2)
-
Hans Hagen -
Thomas A. Schmitz