August 2021 - dev-luatex

Luatex 1.12.0 announcement.
by luigi scarso 27 May '23

27 May '23

============================================================== LuaTeX 1.12.0 2020-03-15 ============================================================== This is the release for TeX live 2020. --shell-restricted now forbids loading of dso (dll/so object) Scaled virtual font Aadditional primitives \eTeXglue[stretch|shrink]order for querying order the etex way. Extra features runtoks tex.getmodenames() (as these can differ per engine/version) Permit nested runtoks in more complex situations of nesting (HH) More accurate and correct test for tex.[set|get]math Minor clean-up and bugs fixed -- luigi

4 8

Fix for pdf_literal Lua string / token list conversions
by Michal Vlasák 07 May '23

07 May '23

Hello, take this plain LuaTeX example: \setbox0=\hbox{\pdfextension literal{0 g}} % 1) \directlua{ local literal = tex.getbox(0).head texio.write_nl("log", "literal.data="..literal.data) } \showbox0 % 2) \directlua{ local literal = tex.getbox(0).head literal.data = "test" literal.token = "toks" texio.write_nl("log", "literal.token="..(literal.token or "nil")) } \showbox0 % 3) patch test \directlua{ tex.set("everyjob", "asd") texio.write_nl("log", tex.get("everyjob")) } \bye Expected log output (abridged): 1) literal.data=0 g .\pdfliteral origin{0 g} 2) literal.token=toks .\pdfliteral origin{toks} Actual output: 1) literal.data=data .\pdfliteral origin{0 g} 2) literal.token=characters .\pdfliteral origin <lua data reference 153> In the first case the Lua accessor returns a value which happens to be on top of the stack (the key "data" itself). In the second case an index into TeX memory is misinterpreted as a Lua registry index, so the returned data is essentially garbage. Patch for both issues is attached. I also extended `tokenlist_from_lua` (which is what `nodelib_gettoks` is defined as) to allow an index argument. The previous version used the value on top of the stack, which probably worked for every use currently in LuaTeX, but seemed rather dangerous and subtle. The patch is from git, though it can be applied normally with patch -Np1 < pdf_literal.patch Kind regards, Michal Vlasák

2 3

Re: [Dev-luatex] bug#48064: texlive-* packages fail to build non-deterministically
by Ludovic Courtès 31 Dec '21

31 Dec '21

Hello, While investigating luatex crashes in the TeX Live 2020 package of GNU Guix¹, we identified the following heap corruption reported by Valgrind (this is on GNU/Linux, with glibc 2.33): --8<---------------cut here---------------start------------->8--- sh-5.0$ ~ludo/.guix-profile/bin/valgrind --extra-debuginfo-path=/gnu/store/f933bvd6ab6l2lg6xl6k1a6jwvcls0z6-glibc-2.33-debug/lib/debug "luatex" "-interaction=nonstopmode" "-output-directory=build" "&luatex" "amsbsy.dtx" ==28531== Memcheck, a memory error detector ==28531== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==28531== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info ==28531== Command: luatex -interaction=nonstopmode -output-directory=build &luatex amsbsy.dtx ==28531== This is LuaTeX, Version 1.12.0 (TeX Live 2020) restricted system commands enabled. ==28531== Invalid write of size 8 ==28531== at 0x485C691: lua_pushlstring (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x568E03: load_hyphenation (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x56B41C: undump_language_data (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4DFB9F: load_fmt_file (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4EF0ED: main_body (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x45118D: main (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== Address 0xae14170 is 0 bytes after a block of size 1,168 alloc'd ==28531== at 0x483EBE1: realloc (in /gnu/store/jlmh0jbgr6zn4iyvhfbvxps8pykzj5ry-valgrind-3.16.1/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==28531== by 0x46695D: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x486D932: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x48660F2: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x4868BE7: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x4868FCF: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x486988B: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x485C6BB: lua_pushlstring (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x568E03: load_hyphenation (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x56B41C: undump_language_data (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4DFB9F: load_fmt_file (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4EF0ED: main_body (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== ==28531== Invalid write of size 4 ==28531== at 0x485C6A2: lua_pushlstring (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x568E03: load_hyphenation (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x56B41C: undump_language_data (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4DFB9F: load_fmt_file (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4EF0ED: main_body (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x45118D: main (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== Address 0xae14178 is 8 bytes after a block of size 1,168 alloc'd ==28531== at 0x483EBE1: realloc (in /gnu/store/jlmh0jbgr6zn4iyvhfbvxps8pykzj5ry-valgrind-3.16.1/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==28531== by 0x46695D: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x486D932: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x48660F2: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x4868BE7: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x4868FCF: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x486988B: ??? (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x485C6BB: lua_pushlstring (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/lib/libtexlua53.so.5.3.5) ==28531== by 0x568E03: load_hyphenation (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x56B41C: undump_language_data (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4DFB9F: load_fmt_file (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== by 0x4EF0ED: main_body (in /gnu/store/w20xxg8p0wksbrxxvj3y46fngvr3954w-texlive-bin-20200406/bin/luatex) ==28531== [...] valgrind: m_mallocfree.c:305 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed. valgrind: Heap block lo/hi size mismatch: lo = 1232, hi = 68. This is probably caused by your program erroneously writing past the end of a heap block and corrupting heap metadata. If you fix any invalid writes reported by Memcheck, this assertion failure will probably go away. Please try that before reporting this as a bug. --8<---------------cut here---------------end--------------->8--- Does that ring a bell? Is there a chance this problem was fixed in the meantime? Thanks in advance, Ludovic. ¹ https://issues.guix.gnu.org/48064

3 4

LuaTeX doesn't discard empty paragraphs when textdir is used
by Marcel Krüger 02 Sep '21

02 Sep '21

Hi, consider the following plain LuaTeX document: \textdir TRT \noindent\par \bye without the `\textdir TRT` line or with `\textdir TLT`, this would lead to "warning (pdf backend): no pages of output.", but with the non default "\textdir", it creates an empty (except for the page number) page instead. This can be avoided by setting \pardir too: \textdir TRT\pardir TRT \noindent\par \bye again produces no output, but adding a group around it reintroduces the issue: \begingroup \textdir TRT\pardir TRT \noindent\par \endgroup \bye leads to an empty page. Of course similar things happen not only for otherwise empty documents: Instead of empty paragraphs disappearing, they add empty lines. Together, this is not only inconsistent with other engines which always discard empty paragraphs, but also leads to hard to predict behavior (especially for users who are not familiar with the implementation of LuaTeX's directional system), so I think it would be great if LuaTeX could always remove such empty paragraphs. Best, Marcel

5 9

Issues with settex() and glue
by Michal Vlasák 28 Aug '21

28 Aug '21

Hello, as part of [1] I also noticed that, the "settex" function (used as metamethod for the "tex" table and for "tex.set") doesn't work for glue specified by (up to) five values case. There are multiple issues. Firstly this is the Lua stack when this function is called (i = lua_gettop()): tex.set("global", "parskip", 1, 2, 3, 4, 5) -> i-6 i-5 i-4 i-3 i-2 i-1 i "global" "parskip" 1 2 3 4 5 The function essentially doesn't do anything for the five number glue case, because of the outer most check: if (lua_type(L, i - 1) == LUA_TSTRING) { Even if it did, it wouldn't respect the "global" setting, because it is checked like this: if (i == 3 && (lua_type(L,1) == LUA_TSTRING)) { const char *s = lua_tostring(L, 1); if (lua_key_eq(s,global)) And even then, it would try to use values beyond the top of the stack: width(value) = lua_roundnumber(L,i); if (i > 1) { stretch(value) = lua_roundnumber(L,i+1); } if (i > 3) { shrink(value) = lua_roundnumber(L,i+2); } if (i > 4) { stretch_order(value) = lua_tointeger(L,i+3); } if (i > 5) { shrink_order(value) = lua_tointeger(L,i+4); } The manual also mentions the 5 value case twice in the same paragraph: --- a/manual/luatex-tex.tex +++ b/manual/luatex-tex.tex @@ -271,9 +271,7 @@ \type {tex.get} you get the width of the glue and when you pass \type {true} you get all five values. Otherwise you get a node which is a copy of the internal value so you are responsible for its freeing at the \LUA\ end. When you set a -glue quantity you can either pass a \nod {glue_spec} or upto five numbers. If -you pass \type {true} to \type {get} you get 5 values returned for a glue and -when you pass \type {false} you only get the width returned. +glue quantity you can either pass a \nod {glue_spec} or upto five numbers. And then the contradicts itself: 10.3.2.4 Glue parameters The glue parameters accept and return a userdata object that represents a glue_spec node. I tried to fix the issues, but the "settex" function is also complicated by the fact, that it also is a __newindex metamethod of the tex table, which means that its stack can also look like this: tex.parskip = n -> i-2 i-1 i tex "parskip" n So one can't easily just index from the the bottom. Also the line: int a = isglobal; in the glue case would deserve a comment, since it is subtle, but required for define() to work properly (it looks at the variable called "a"). I would prefer something like: int a = isglobal; /* for define() */ In the end I am not even sure how to handle the issues. I leave it to the more experienced. Kind regards, Michal Vlasák [1]: https://mailman.ntg.nl/pipermail/dev-luatex/2021-August/006542.html

1 0

lastnodetype, again
by Robert 22 Aug '21

22 Aug '21

Hi, while the last problem with \lastnodetype (cf. https://mailman.ntg.nl/pipermail/dev-luatex/2020-April/006414.html) has been fixed, there are still incompatibilities if it is preceded by \relax. The following outputs 7,0,8,7,7,7 with pdftex and etex, but 7,0,0,0,0,0 with luatex: %\input luaotfload.sty %\font\x=[lmroman12-regular.otf]:script=latn;+trep;+tlig;%mode=base; %\x % no difference \setbox0\hbox{--} \unhbox0 \the\lastnodetype % OK --\the\lastnodetype % OK --\relax\the\lastnodetype \setbox0\hbox{--\relax\the\lastnodetype} \unhbox0 \toks0={--} \setbox0\hbox{\the\toks0\relax \the\lastnodetype} \unhbox0 \setbox0\vbox{\hbox{--\relax\the\lastnodetype}} \unvbox0 \bye Best, -- Robert

2 4

harfbuzz 2.8.2
by luigi scarso 18 Aug '21

18 Aug '21

I have pushed harfbuzz 2.8.2 into experimental, if it's ok I will sync with the texlive repo. -- luigi

1 0

Improvements for glyph_stream_provider with TTF fonts
by Marcel Fabian Krüger 09 Aug '21

09 Aug '21

Hi, while working with the glyph_stream_provider callback I noticed that it has some issues, especially around TTF fonts. Especially LuaTeX analyzes the glyph streams *before* the callback get involved to identify composite glyphs. If composite glyphs are found, the corresponding components get added to the subset bug get new glyph ids. Later, when glyph_stream_provider is called to get the glyph streams, the parameters are the new glyph ids which are assigned by LuaTeX and the mapping is not made available. This leads to two issues: - When glyph_stream_provider is used, composite glyphs can't be preserved but have to be expanded since the callback has no way of knowing the final (remapped) glyph ids of the components. This can not only make the file bigger, it can also invalidate hinting instructions. - Even though the remapped components can't be found from the composite glyphs, they still trigger a call to the callback which can't be distinguished from normal calls and therefore still lead to something being added to the embedded font. Since only the remapped GID is made available, this will not even be the correct component glyph which happens to be in the slot described by the glyph id the component got mapped to (typically glyphs 1, 2, etc. since the remapping maps to the smallest unused glyph ids). This mostly just makes the embedded font bigger and it's confusing when looking at these subsetted fonts with weird glyphs included in the subset. - Somewhat independently, this current code does not allow the side bearings and advance widths to be modified which means that some table values can get inconsistent for variable fonts which change these values. This this is used by existing code and mostly just causes inefficiencies and not outright errors, the existing behavior should probably be kept available for compatibility, so I suggest the following addition: When a font has font_streamprovider set to 3, behave exactly as if the streamprovider was 2, except that before the "normal" glyph_stream_provider provider calls for a font happen (with arguments fontid, (remapped) gid, 2) call the `glyph_stream_provider` once for every glyph with the four arguments (fontid, remapped gid, 3, original gid) and expect four return values, corresponding to horizontal advance width, left side bearing, vertical advance width and top side bearing. This ensures that when the glyphstreams are generated, the callback code is aware of the full mapping between original and remapping GIDs and can therefore use these remapped GIDs correctly. An example implementation is attached. Best regards, Marcel

2 7

[PATCH] luaharfbuzz: Provide interface for variable fonts
by Marcel Fabian Krüger 04 Aug '21

04 Aug '21

Hi, please consider the attached patch for luahbtex. It adds a Lua interface for HarfBuzz's functions around variable fonts. It is an adoption of recent upstream luaharfbuzz changes with additional LuaJIT support. While corresponding busted tests are included too, the font needed to actually run these tests has not been added to the fonts directory since sending binary files in diffs via email is problematic and I don't think that anyone is running these tests from the LuaTeX repo anyway. Best regards, Marcel

2 7