- dev-luatex - NTG Mailing Lists

reproducibility issue of pdf inclusion of LuaTeX 1.09.0
by Hironobu Yamashita 07 Dec '18

07 Dec '18

Hi, I'm suffering from reproducibility issue of pdf inclusion of LuaTeX. The output file size differs from time to time, and the largest one is more than twice of the smallest one! Please find the attached 20181203-luatex-pdfobj.tar.xz ======== Details: ======== My binaries are based on TeX Live svn r49302 (self-build on macOS) and I am using LuaTeX with Lua5.2: $ luatex --version This is LuaTeX, Version 1.09.0 (TeX Live 2019/dev) [.. snip ..] LuaTeX is Copyright 2018 Taco Hoekwater and the LuaTeX Team. With the attached source, I run $ export SOURCE_DATE_EPOCH=0 $ export FORCE_SOURCE_DATE=1 $ for i in `seq 0 256`; do lualatex -jobname=include-$i include; done (SOURCE_DATE_EPOCH and FORCE_SOURCE_DATE are set above, to ensure that all output binaries are expected to be identical) The output file (include-[num].pdf) sizes differ from time to time as follows: [size] - [frequency] 63928 bytes - 230 63947 bytes - 1 63958 bytes - 1 110923 bytes - 2 110924 bytes - 10 110925 bytes - 2 110928 bytes - 1 110931 bytes - 1 110932 bytes - 1 110935 bytes - 7 157913 bytes - 1 The largest one is more than twice of the smallest one (= the most frequent one)! Attached are some representatives, and I noticed that the log files also show some strange difference of object numbers: * include-0: 63928 bytes PDF statistics: 72 PDF objects out of 1000 (max. 8388607) 36 compressed objects within 1 object stream * include-11: 110924 bytes PDF statistics: 86 PDF objects out of 1000 (max. 8388607) 44 compressed objects within 1 object stream * include-201: 158913 bytes PDF statistics: 100 PDF objects out of 1000 (max. 8388607) 52 compressed objects within 1 object stream I never experienced this kind of difference with LuaTeX 1.07.0 of TeX Live 2018. Could you examine this issue? Thanks -- Hironobu ---- 山下弘展 (Hironobu Yamashita) e-mail: h.y.acetaminophen(a)gmail.com

3 7

Re: [Dev-luatex] reproducibility issue of pdf inclusion of LuaTeX
by Ulrike Fischer 03 Dec '18

03 Dec '18

(sorry but gmane seems not to work for this list, so I can't send proper "answers") > The output file (include-[num].pdf) sizes differ from time to time > as follows: I can reproduce the issue with Development id: 6871. I looked into the uncompressed pdf and made a diff found that some font resources are sometimes twice in the pdf: 13 0 obj ... %%BeginResource: CMap (TeX-CJZVFX-LibertinusSerif-0) %%Title: (TeX-CJZVFX-LibertinusSerif-0 TeX CJZVFX-LibertinusSerif 0) 63 0 obj ... %%BeginResource: CMap (TeX-CJZVFX-LibertinusSerif-0) %%Title: (TeX-CJZVFX-LibertinusSerif-0 TeX CJZVFX-LibertinusSerif 0) -- Mit freundlichen Grüßen Ulrike Fischer mailto:luatex@nililand.de

1 0

\pdfextension startlink / endlink and boxing level problems
by Ulrike Fischer 01 Dec '18

01 Dec '18

(A similar question has been sent to the tex live list for pdftex) After some discussion in the latex team I have two questions about \pdfextension startlink/endlink in luatex and boxing levels. 1. Currently the commands give a fatal error if used in different boxing levels: x\hbox{\pdfextension startlink attr {/Border[0 0 1]} user{ /Subtype/Link% /A<<% /Type/Action% /S/URI% /URI(blkub)% >>% } Linktext} \pdfextension endlink \bye ! error: (pdf backend): 'endlink' ended up in different nesting level than 'startlink' ! ==> Fatal error occurred, no output PDF file produced! This doesn't happen very often in documents, but if the galley is split and reboxed it is possible that long links end in different levels. One example involving two column mode and top floats has been discussed on the latex2e github: https://github.com/latex3/latex2e/issues/94 Is the fatal error really sensible here? Couldn't the engine close the last annotation with some more or less sensible coordinate and continue with a warning? A pdf with a wrong link area seems better than no pdf at all (and would release formats and packages from the need to carefully keep track of boxing levels when splitting and reassembling stuff). 2. The engine interrupts linking if the boxing level of intermediate text is less than the boxing level at the start of the link: x\hbox{\hbox{\pdfextension startlink attr {/Border[0 0 1]} user{ /Subtype/Link% /A<<% /Type/Action% /S/URI% /URI(blkub)% >>% } Linktext}} not linked \hbox{\hbox{still not linked} \hbox{\hbox{more Linktext\pdfextension endlink}} \bye This allows to some extend to exclude e.g. header and footer from link areas. But doesn't work e.g. with the latex package fancyhdr which adds an additional box level to the footer. xetex/xdvidfpmx has here a quite useful special \special{pdf:nolink} which can be used to suppress the unwanted link area: \documentclass[a4paper]{article} \usepackage{fancyhdr} \pagestyle{fancy} \cfoot{some longer text that should not be linked} \usepackage{ifxetex} \ifxetex %\cfoot{\special{pdf:nolink}some longer text that should not be linked\special{pdf:link}} \fi \usepackage{hyperref} \begin{document} \vspace*{48\baselineskip} xxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxx \href{xxxxx}{link link link link } \end{document} Would it be possible to add to luatex something similar? -- Ulrike Fischer http://www.troubleshooting-tex.de/

2 1

Error reading fonts with version 0 OS/2 table
by Khaled Hosny 27 Nov '18

27 Nov '18

Hi all, LuaTeX stops with an error when reading version 0 OS/2 table, and this is is because it unconditionally reads ulCodePageRange1 and ulCodePageRange2 fields that were not defined in version 0. Such fonts can be found here [1] (from the question [2]). I’m attaching a patch that makes this conditional, it also fixed reading version 2 specific fields that would be skipped for tables with versions greater than 2. It does not look like these fields are used anywhere, so another fix is to only read the fields LuaTeX actually needs. Regards, Khaled 1. https://mp.uu.se/documents/432512/911262/18350_Typsnitt_PC.zip/4c0954cf-84b… 2. https://tex.stackexchange.com/q/430837/729

1 0

Recalculating tail of the list in ligaturing callback
by Esger Renkema 16 Nov '18

16 Nov '18

Dear list, Running the following one-liner: \directlua{callback.register('ligaturing', node.ligaturing)}ffi\bye results in: ------------------------------------------- This is LuaTeX, Version 1.09.0 (TeX Live 2019/dev/Debian) restricted system commands enabled. **\directlua{callback.register('ligaturing', node.ligaturing)}ffi\bye ! error: (linebreak): invalid list tail, probably missing glue ! ==> Fatal error occurred, no output PDF file produced! ------------------------------------------- The problem seems to be that, when restoring the original ‘tail of the list’ after applying the ligaturing callback (in this case consisting of the parfillskip glue), it is appended to the tail of the /original/ list-to-be-ligatured. The latter, of course, has been lost if the paragraph ends in a ligature. According to the manual, the first node of the list passed to the callback is guaranteed not to be a glyph_node. Should the same not hold for the last one, too? The manual states that it ‘normally’ is a glue, but that is clearly not the case here. Best, Esger

3 5

Reading tounicode from shared fonts
by Khaled Hosny 14 Nov '18

14 Nov '18

Hi, While debugging an issue with PDF /ToUnicode, I found code for reading character-level tounicode from the fonts, always takes them from the first font in fonts in merges in the PDF. Usually this is not an issue since the font loader will have identical tounicode values for all characters in the same font loaded multiple times. However, my code sets character-level tounicode only after processing the nodes (to avoid parsing the cmap and GSUB tables ahead of typesetting), so the same font loaded multiple times can have different tounicodes and characters used only in later instances of the font will not have their tounicodes in the PDF file. It seems to be one character fix to instead check the font the character is used in, and looks to me like a typo since the first revision this code was introduced (r710) that went unnoticed. Patch attached. Regards, Khaled

2 1

255 characters limit on font names
by Khaled Hosny 13 Nov '18

13 Nov '18

Hi all, Currently fonts with names are longer than 255 characters are ignored and the font fails to load without an explanation. See https://github.com/lualatex/luaotfload/issues/424 for background. The code suggests this is a limitation of the DVI format (the relevant check was add back in 2009 according to repository history), but it shouldn’t affect PDF mode. The attached patch restricts the check to DVI mode and adds an error message. Alternatively the font name can be just truncated in DVI mode (which what happens if the check is removed, but the extra characters are removed from the start of the name, no idea why) an just issue a warning, or use the file name for DVI format since the rest of the spec is irrelevant to DVI files (but this can be debatable, I don’t know if there are any DVI drivers that can parse LuaTeX font names and load the fonts). Regards, Khaled

2 3

Re: [Dev-luatex] Callbacks in Lua wrapper of mplib
by Marcel Krüger 08 Nov '18

08 Nov '18

---- On Wed, 07 Nov 2018 13:00:50 +0100 luigi scarso <luigi.scarso(a)gmail.com> wrote ---- > > > On Wed, Nov 7, 2018 at 12:18 PM Marcel Krüger <tex(a)2krueger.de> wrote: > Hi, > > I recently discovered that the Lua wrapper of mplib registers all callbacks globally > such that creating multiple mplib instances with different callbacks breaks. > > > why not this ? > \directlua{ local current_instance local custom_find_file = custom_find_file or {} local mp_find_file_driver = function(name, mode, type) print("I am ",current_instance) if custom_find_file[current_instance] then return custom_find_file[current_instance](name, mode, type) else return "unregistered" end end custom_find_file['mp1'] = function(name, mode, type) return name end custom_find_file['mp2'] = function(name, mode, type) if name == 'cmr10' then print'This should not happen' end return name end > local mp1 = mplib.new{ find_file = mp_find_file_driver } > local mp2 = mplib.new{ find_file = mp_find_file_driver } current_instance = "mp1" mp1:execute[[ input cmr10; ]] > current_instance = "mp2" mp2:execute[[ input cmr10; ]] > > } \bye That's similar to the workaround I'm using right now, but it seems odd: The callbacks are given as in the initializer table of a single instance, so you do not expect them to affect other instances. Especially this means that if anyone uses "short-living" MPLib instances, this person will probably not expect this behaviour and just pass the find_file method directly. This means that every user of "long-living" MPLib instances has to fear that every called library might change this global state. If this happens, there does not even seem to be a way to change it back, so the safest approach would be to call mplib.new and create and destroy a temporary instance every time `:execute` or `:finish` is called. So if this is the intended behaviour, IMO a note should be added to the documentation. Especially because every non-function entry in the initializer only affects this instance. But I still think it should be changed: The current behaviour seems unintuitive and inconsistant. Also manually keeping track of the current instance seems at least odd if mplib can access this information anyway. Additionally the change would make the Lua API of mplib more similar to the C API. Best regards Marcel Krüger

2 1

Callbacks in Lua wrapper of mplib
by Marcel Krüger 07 Nov '18

07 Nov '18

Hi, I recently discovered that the Lua wrapper of mplib registers all callbacks globally such that creating multiple mplib instances with different callbacks breaks. This affects the undocumented `run_script` and `make_text` callbacks, but also the documented `find_file`: For example the plain LuaTeX document \directlua{ local mp = mplib.new{ find_file = function(name, mode, type) return name end } do % Create a temporary second instance with a different run_script handler mplib.new{ find_file = function(name, mode, type) if name == 'cmr10' then print'This should not happen' end return name end }:finish() end % Now trigger find_file using the *first* mp instance which should *not* trigger a message mp:execute[[ input cmr10; ]] } \bye shows the message 'This should not happen', because the find_file implementation of the second call to `mplib.new` is also used for the first `mp` instance. This can be fixed by storing the callbacks in userdata associated with the mp uservalue instead of in the global registry: diff --git a/source/texk/web2c/mplibdir/lmplib.c b/source/texk/web2c/mplibdir/lmplib.c index eb8432bb5..88873ed66 100644 --- a/source/texk/web2c/mplibdir/lmplib.c +++ b/source/texk/web2c/mplibdir/lmplib.c @@ -329,7 +329,9 @@ static char *mplib_find_file(MP mp, const char *fname, const char *fmode, int ft { lua_State *L = (lua_State *)mp_userdata(mp); lua_checkstack(L, 4); - lua_getfield(L, LUA_REGISTRYINDEX, "mplib.file_finder"); + lua_getuservalue(L, 1); + lua_getfield(L, -1, "file_finder"); + lua_replace(L, -2); if (lua_isfunction(L, -1)) { char *s = NULL; const char *x = NULL; @@ -365,9 +367,9 @@ static int mplib_find_file_function(lua_State * L) /*tex An error. */ return 1; } - lua_pushstring(L, "mplib.file_finder"); + lua_pushstring(L, "file_finder"); lua_pushvalue(L, -2); - lua_rawset(L, LUA_REGISTRYINDEX); + lua_rawset(L, -4); return 0; } @@ -379,8 +381,10 @@ static void mplib_warning(const char *str) static void mplib_script_error(MP mp, const char *str) { lua_State *L = (lua_State *)mp_userdata(mp); - lua_checkstack(L, 1); - lua_getfield(L, LUA_REGISTRYINDEX, "mplib.script_error"); + lua_checkstack(L, 2); + lua_getuservalue(L, 1); + lua_getfield(L, -1, "script_error"); + lua_replace(L, -2); if (lua_isfunction(L, -1)) { lua_pushstring(L, str); /*tex We assume that the function is okay. */ @@ -397,17 +401,19 @@ static int mplib_script_error_function(lua_State * L) /*tex An error. */ return 1; } - lua_pushstring(L, "mplib.script_error"); + lua_pushstring(L, "script_error"); lua_pushvalue(L, -2); - lua_rawset(L, LUA_REGISTRYINDEX); + lua_rawset(L, -4); return 0; } static char *mplib_run_script(MP mp, const char *str) { lua_State *L = (lua_State *)mp_userdata(mp); - lua_checkstack(L, 1); - lua_getfield(L, LUA_REGISTRYINDEX, "mplib.run_script"); + lua_checkstack(L, 2); + lua_getuservalue(L, 1); + lua_getfield(L, -1, "run_script"); + lua_replace(L, -2); if (lua_isfunction(L, -1)) { char *s = NULL; const char *x = NULL; @@ -433,17 +439,19 @@ static int mplib_run_script_function(lua_State * L) if (!(lua_isfunction(L, -1) || lua_isnil(L, -1))) { return 1; /* error */ } - lua_pushstring(L, "mplib.run_script"); + lua_pushstring(L, "run_script"); lua_pushvalue(L, -2); - lua_rawset(L, LUA_REGISTRYINDEX); + lua_rawset(L, -4); return 0; } static char *mplib_make_text(MP mp, const char *str, int mode) { lua_State *L = (lua_State *)mp_userdata(mp); - lua_checkstack(L, 1); - lua_getfield(L, LUA_REGISTRYINDEX, "mplib.make_text"); + lua_checkstack(L, 2); + lua_getuservalue(L, 1); + lua_getfield(L, -1, "make_text"); + lua_replace(L, -2); if (lua_isfunction(L, -1)) { char *s = NULL; const char *x = NULL; @@ -471,9 +479,9 @@ static int mplib_make_text_function(lua_State * L) /*tex An error. */ return 1; } - lua_pushstring(L, "mplib.make_text"); + lua_pushstring(L, "make_text"); lua_pushvalue(L, -2); - lua_rawset(L, LUA_REGISTRYINDEX); + lua_rawset(L, -4); return 0; } @@ -544,6 +552,8 @@ static int mplib_new(lua_State * L) /* options->script_error = mplib_script_error; */ options->print_found_names = 1; options->ini_version = 1; + /*tex Here we create the uservalue table for the callbacks: */ + lua_newtable(L); if (lua_type(L, 1) == LUA_TTABLE) { for (i = 0; mplib_parms[i].name != NULL; i++) { lua_getfield(L, 1, mplib_parms[i].name); @@ -606,6 +616,7 @@ static int mplib_new(lua_State * L) lua_pop(L, 1); } } + lua_setuservalue(L, -2); *mp_ptr = mp_initialize(options); xfree(options->command_line); xfree(options->mem_name);

2 1

Fwd: Re: values for `mode` field in tex.nest
by Hans Hagen 29 Oct '18

29 Oct '18

On 10/28/2018 2:53 PM, Marcel Krüger wrote: > Hi, > > according to the documentation, the `mode` field in `tex.nest` is explained as > >> a number representing the main mode at this level: 0 = no >> mode (this happens during \write), 1 = vertical, 127 = hor- >> izontal, 253 = display math, -1 = internal vertical, -127 = >> restricted horizontal, -253 = inline math > > This looks as if these values were constant across engine versions, but the real values > depend on `max_command_cmd` which changes if new commands are added. > Especially the current values are `(-)132` for (restricted) horizontal mode and > `(-)263` for (inline) math. > > So I think the documentation should be either updated to delete these outdated values > and instead include a note that you have to query these values yourself or LuaTeX could > normalize these values before passing them to Lua. > > One way to do this could be i'll give them names (hmode vmode innerhmode innervmode inlinemath displaymath none) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

1 0