- dev-luatex - NTG Mailing Lists

Wrong spacing for OpenType math subscripts
by Johannes Rosenberger 15 Mar '21

15 Mar '21

Hi LuaTeX developers, first of all, thank you very much for all your work on this and the great OpenType support and possibility of fixing internals using Lua code! In the following document, the math subscripts are so close to the following characters that they sometimes overlap: \documentclass{article} \usepackage{unicode-math} \def\supp{{\rm supp}} \begin{document} $$ I(P_X,P_{Y|X}) = \sum_{(X,Y) \in \supp P_{XY}} P_{XY}(X,Y) \log_2 \frac{P_{XY}(X,Y)}{P_X(X) P_Y(Y)}. $$ \end{document} The problem cannot be fixed by simply adding spacing, e.g. \Uordopenspacing, because the space is different per character. We need italics correction, in all circumstances where an italic symbol is last in the subscript and something follows the noad. By the manual, Section 7.6.4 this is not applied to subscripts, but only the base characters. Setting \mathitalicsmode=1 to force it changes nothing, because italics correction is then only applied in the subscript to the characters before the last one. Maybe luatex assumes the last character to be followed by nothing, because in the mlist of the subscript follows nothing even if after the whole noad follows something. Setting \mathitalicsmode=1 and adding manual 0-kerns (like \/) helps. This is implemented by the following lua code: \directlua{ ty = node.types() sub_addkern = function(sub) scan_sub_addkern(sub) local kern = node.new("kern") if ty[sub.id] == "math_char" or ty[sub.id] == "math_char_text" then local old = node.new("noad", 0) old.nucleus = sub sub = node.new("sub_mlist") sub.head = old node.insert_after(sub.head, old, kern) elseif ty[sub.id] == "sub_box" or ty[sub.id] == "sub_mlist" then local tail = node.tail(sub.head) if not (ty[tail.id] == "kern") then node.insert_after(sub.head, tail, kern) end end return sub end scan_sub_addkern = function(n) if n.head then n.head = scan_all_sub_addkern(n.head) end if n.sub then n.sub = sub_addkern(n.sub) end if n.nucleus then scan_sub_addkern(n.nucleus) end if n.num then scan_sub_addkern(n.num) end if n.denom then scan_sub_addkern(n.denom) end end scan_all_sub_addkern = function(head) for n in node.traverse(head) do scan_sub_addkern(n) end return head end luatexbase.add_to_callback("pre_mlist_to_hlist_filter", scan_all_sub_addkern, "default mlist_to_hlist + italics-corrected subscript") } XeTeX does this right out of the box. Best, Johannes

3 4

[fs]io.readinteger4 produces excessively small negative values
by Marcel Fabian Krüger 07 Mar '21

07 Mar '21

Hi, Look at the texlua script: print(sio.readinteger4(string.pack(">i", -1), 1)) It would be expected that this prints "-1", but at least is should print a number in the range of 4-byte signed integers (-2147483648 -- 2147483647). Instead it prints -4294967297. The same issue appears with fio.readinteger4 and [fs]readintegertable(..., 4): Whenever the value is negative, the returned value is 2^32 lower than expected. (Technically this only happens if Lua's integer type allows at least 64-bit integers and the native "int" type has at most 32-bit, but this is the case on basically any current platform. The cause is code like the following in the affected C implementations: int a = ...; int b = ...; int c = ...; int d = ...; if (a >= 0x80) lua_pushinteger(L, 0x1000000 * a + 0x10000 * b + 0x100 * c + d - 0x100000000); else ...; Here a, b, c, d are "int", so according to C language rules "0x1000000 * a + 0x10000 * b + 0x100 * c + d" is evaluated as a (signed) int. If a>=0x80 then 0x1000000*0x80 is bigger than 2^31 and can therefore not be represented in a (32-bit) int. Theoretically this triggers undefined behavior since LuaTeX does not compile with "-fwrapv" or similar, but on my system it does "the expected thing" and wraps. Therefore the value is already negative and the subtraction of 0x100000000, which is evaluated in 64bit since the literal can not be represented in 32bit, leads to the unexpected value. This can be fixed by making sure that the expression is completely evaluated with 64-bit integers (or whatever integer type Lua uses). A possible patch is attached. Best regards, Marcel

3 2

Including PDF images with page /Metadata streams
by Marcel Fabian Krüger 01 Mar '21

01 Mar '21

Hi, (corresponding bug report for LaTeX: https://tex.stackexchange.com/questions/585289/corrupt-pdf-with-includegrap… ) when LuaTeX includes a PDF file in which a Page object contains a Metadata stream, then the output is a invalid PDF file since LuaTeX adds the stream inline into the directory instead of adding a reference. plain LuaTeX example: Download page.pdf from https://bitbucket.org/longwayahead/latex-problem/src/master/ and then compile with plain LuaTeX: \saveimageresource{./page.pdf} \useimageresource\lastsavedimageresourceindex \bye The issue can be fixed by removing a single character: In pdftoepdf.c, line 875 a comment correctly mentions that indirection has to be preserved, but for this to work the previous ppdict_rget_obj has to be replaced with ppdict_get_obj to ensure that indirections are not already resolved. Best regards, Marcel diff --git a/source/texk/web2c/luatexdir/image/pdftoepdf.c b/source/texk/web2c/luatexdir/image/pdftoepdf.c index 78dab3a37..ca68ba8dc 100644 --- a/source/texk/web2c/luatexdir/image/pdftoepdf.c +++ b/source/texk/web2c/luatexdir/image/pdftoepdf.c @@ -864,15 +864,15 @@ void write_epdf(PDF pdf, image_dict * idict, int suppress_optional_info) if (obj != NULL && obj->type != PPREF) { formatted_warning("pdf inclusion","/Metadata must be indirect object"); } /* copy selected items in Page dictionary */ for (i = 0; pagedictkeys[i] != NULL; i++) { - obj = ppdict_rget_obj(pageDict, pagedictkeys[i]); + obj = ppdict_get_obj(pageDict, pagedictkeys[i]); if (obj != NULL) { pdf_add_name(pdf, pagedictkeys[i]); /* preserves indirection */ copyObject(pdf, pdf_doc, obj); }

2 1

[PATCH] Nested discretionaries
by Marcel Fabian Krüger 18 Feb '21

18 Feb '21

Hi, as documented in the manual, LuaTeX supports a restricted form of nested discretionaries to handle cases like "of\-f\-ice" with ffi/ff/fi ligatures. Sadly this does not seem to work correctly. Take for example the document: \pretolerance-1 \parfillskip0pt \setbox0\hbox{off-} \setbox2\hbox{ice} \noindent \parshape 2 0pt \wd0 0pt \wd2 off\-ice \noindent \parshape 2 0pt \wd0 0pt \wd2 of\-f\-ice \setbox0\hbox{of-} \setbox1\hbox{fice} \noindent \parshape 2 0pt \wd0 0pt \wd1 of\-fice \noindent \parshape 2 0pt \wd0 0pt \wd1 of\-f\-ice \setbox1\hbox{f-} \noindent \parshape 3 0pt \wd0 0pt \wd1 0pt \wd2 of\-f\-ice \bye Here the first two paragraphs are formed to prefectly fit off-ice and this break works correctly when the input is off\-ice. But for of\-f\-ice, no break occurs and the line becomes overfull. The next two paragraphs fit to of-fice, which again works for of\-fice. But for of\-f\-ice, it is broken as off\-ice instead, leading to a overfull followed by a underfull box. The last paragraph tries to trigger both breaks and doesn't work correctly either. The cause are multiple issues: 1. In ext_do_line_break, the disc_width for the second case is for the case that both breaks occur, which is significantly less likely than the single break case. (Since the middle part will normally be much shorter than a paragraph) This can be fixed by switching to the disabled code which according to a comment "doesn't work". It can be fixed by adding a do_one_seven_eight(add_disc_width_to_active_width); 2. In compute_break_width, the break_width for a select_disc is not handled specially. Therefore instead of subtracting the no_break width of the previous discretionary which is the width which is contributed for the construct to active_width. 3. After both changes, the only remaining issue occurs when both breaks trigger at the same time. Since the calculations assume the other case, the length of the middle line is calculated incorrectly. Since no glue can appear in discretionary nodes, we know that this error can not affect the stretchable and shrikable component of the line, so we can just overwrite the natural width with the statically known length of this line to fix this. IMO the thirs issue is much less important than the first two since it only affects the corner-case of taking both breaks. The diff below fixes all three issues and additionally allows the disc_penalty of the select_disc to determine the penallty of the second break. (That was useful during debugging and it seems like the expected behavior.) I also attach three diffs with more or less independent components: First only the penalty change, than a change for 1&2 which fixes most common issue and then the slightly more hacky fix for 3. Best regards, Marcel diff --git a/source/texk/web2c/luatexdir/tex/linebreak.c b/source/texk/web2c/luatexdir/tex/linebreak.c index 646a5a771..8c66dbf32 100644 --- a/source/texk/web2c/luatexdir/tex/linebreak.c +++ b/source/texk/web2c/luatexdir/tex/linebreak.c @@ -879,7 +879,8 @@ static int line_diff; double_hyphen_demerits, \ final_hyphen_demerits, \ first_p, \ - cur_p \ + cur_p, \ + null \ ); \ if (type(cur_p)!=math_node) \ active_width[1] += width(cur_p); \ @@ -1094,7 +1095,10 @@ static void compute_break_width(int break_type, int line_break_dir, int adjust_s path, as we are talking about the breaking on {\it this} position. */ - sub_from_widths(vlink_no_break(p), line_break_dir, adjust_spacing, break_width); + if (subtype(p) == select_disc) + sub_from_widths(vlink_no_break(alink(p)), line_break_dir, adjust_spacing, break_width); + else + sub_from_widths(vlink_no_break(p), line_break_dir, adjust_spacing, break_width); add_to_widths(vlink_post_break(p), line_break_dir, adjust_spacing, break_width); do_one_seven_eight(add_disc_width_to_break_width); if (vlink_post_break(p) == null) { @@ -1273,7 +1277,8 @@ static void ext_try_break( int line_penalty, int last_line_fit, int double_hyphen_demerits, - int final_hyphen_demerits, halfword first_p, halfword cur_p + int final_hyphen_demerits, halfword first_p, halfword cur_p, + halfword init_p ) { /*tex runs through the active list */ @@ -1311,6 +1316,8 @@ static void ext_try_break( scaled g = 0; /*tex distance from current active node */ scaled cur_active_width[10] = { 0 }; + /*tex if cur_p == init_p != null, the line length between init_disc and select_disc */ + scaled init_select_width; /*tex Make sure that |pi| is in the proper range; */ if (pi >= inf_penalty) { /*tex this breakpoint is inhibited by infinite penalty */ @@ -1371,6 +1378,11 @@ static void ext_try_break( no_break_yet = false; do_all_eight(set_break_width_to_background); compute_break_width(break_type, line_break_dir, adjust_spacing, cur_p); + if (init_p != null && init_p == cur_p) { + scaled line_width[10] = { 0 }; + add_to_widths(vlink_pre_break(vlink(cur_p)), line_break_dir, adjust_spacing, line_width); + init_select_width = background[1] + line_width[1]; + } } /*tex @@ -1448,6 +1460,8 @@ static void ext_try_break( active_short(q) = best_pl_short[fit_class]; active_glue(q) = best_pl_glue[fit_class]; } + if (init_p != null && init_p == cur_p) + active_select_width(q) = init_select_width; vlink(q) = r; vlink(prev_r) = q; prev_r = q; @@ -1513,7 +1527,10 @@ static void ext_try_break( */ artificial_demerits = false; - shortfall = line_width - cur_active_width[1]; + if (init_p != null && break_node(r) != null && cur_break(break_node(r)) == init_p) + shortfall = line_width - active_select_width(r); + else + shortfall = line_width - cur_active_width[1]; if (break_node(r) == null) shortfall -= init_internal_left_box_width; else @@ -2233,7 +2250,8 @@ void ext_do_line_break( double_hyphen_demerits, final_hyphen_demerits, first_p, - cur_p + cur_p, + null ); } } @@ -2285,7 +2303,7 @@ void ext_do_line_break( tracing_paragraphs, protrude_chars, line_penalty, last_line_fit, double_hyphen_demerits, - final_hyphen_demerits, first_p, cur_p); + final_hyphen_demerits, first_p, cur_p, null); } else { add_to_widths(s, line_break_dir, adjust_spacing, disc_width); do_one_seven_eight(add_disc_width_to_active_width); @@ -2295,45 +2313,35 @@ void ext_do_line_break( tracing_paragraphs, protrude_chars, line_penalty, last_line_fit, double_hyphen_demerits, - final_hyphen_demerits, first_p, cur_p); + final_hyphen_demerits, first_p, cur_p, + subtype(cur_p) == init_disc ? cur_p : null); if (subtype(cur_p) == init_disc) { /*tex - We should at two break points after the one - we added above: - - \startitemize[n] - \startitem - which does a possible break in INIT's - |post_break| - \stopitem - \startitem - which means the |no_break| actually - was broken just a character later - \stopitem - \stopitemize - - Do the select-0 case |f-f-i|: - + The breakpoint after the one we added above + is special since it's corresponding + disc_width depends on whether the previous + breakpoint will be taken. Since this would need + adjustments in multiple places, we avoid it by + exploiting the fact that we actually know how long + the line is going to be in that case: The length of + the background plus the SELECT's |pre_break|. + This length is stored in the active nodes for the + INIT since this allows to avoid the computation if + there is no feasible break at INIT. + To trigger this mechanism, we have to pass the + corresponding INIT to ext_try_break. */ - s = vlink_pre_break(vlink(cur_p)); - add_to_widths(s, line_break_dir, adjust_spacing, disc_width); - ext_try_break(actual_penalty, hyphenated_node, - line_break_dir, adjust_spacing, - par_shape_ptr, adj_demerits, - tracing_paragraphs, - protrude_chars, line_penalty, - last_line_fit, double_hyphen_demerits, - final_hyphen_demerits, first_p, - vlink(cur_p)); - /*tex This does not work. */ -#if 0 + actual_penalty = (int) disc_penalty(vlink(cur_p)); /*tex Go back to the starting situation. */ do_one_seven_eight(sub_disc_width_from_active_width); do_one_seven_eight(reset_disc_width); + /*tex Ensure that alink is set for |compute_break_width| */ + alink(vlink(cur_p)) = cur_p; /*tex Add select |no_break| to |active_width|. */ s = vlink_no_break(vlink(cur_p)); add_to_widths(s, line_break_dir, adjust_spacing, disc_width); + do_one_seven_eight(add_disc_width_to_active_width); ext_try_break(actual_penalty, hyphenated_node, line_break_dir, adjust_spacing, par_shape_ptr, adj_demerits, @@ -2341,8 +2349,7 @@ void ext_do_line_break( protrude_chars, line_penalty, last_line_fit, double_hyphen_demerits, final_hyphen_demerits, first_p, - vlink(cur_p)); -#endif + vlink(cur_p), cur_p); } do_one_seven_eight(sub_disc_width_from_active_width); } @@ -2356,7 +2363,7 @@ void ext_do_line_break( tracing_paragraphs, protrude_chars, line_penalty, last_line_fit, double_hyphen_demerits, final_hyphen_demerits, - first_p, cur_p); + first_p, cur_p, null); break; case boundary_node: case whatsit_node: @@ -2398,7 +2405,7 @@ void ext_do_line_break( adjust_spacing, par_shape_ptr, adj_demerits, tracing_paragraphs, protrude_chars, line_penalty, last_line_fit, double_hyphen_demerits, - final_hyphen_demerits, first_p, cur_p); + final_hyphen_demerits, first_p, cur_p, null); if (vlink(active) != active) { /*tex Find an active node with fewest demerits; */ r = vlink(active); diff --git a/source/texk/web2c/luatexdir/tex/texnodes.h b/source/texk/web2c/luatexdir/tex/texnodes.h index 9742ca496..3a1c5ff77 100644 --- a/source/texk/web2c/luatexdir/tex/texnodes.h +++ b/source/texk/web2c/luatexdir/tex/texnodes.h @@ -976,13 +976,14 @@ typedef enum { /* node sused in the parbuilder */ -# define active_node_size 4 /*number of words in extended active nodes */ +# define active_node_size 5 /*number of words in extended active nodes */ # define fitness subtype /*|very_loose_fit..tight_fit| on final line for this break */ # define break_node(a) vlink((a)+1) /*pointer to the corresponding passive node */ # define line_number(a) vinfo((a)+1) /*line that begins at this breakpoint */ # define total_demerits(a) varmem[(a)+2].cint /* the quantity that \TeX\ minimizes */ # define active_short(a) vinfo(a+3) /* |shortfall| of this line */ # define active_glue(a) vlink(a+3) /*corresponding glue stretch or shrink */ +# define active_select_width(a) vlink(a+4) /*if this break is a init_disc, the width of the following line when broken at select_disc */ # define passive_node_size 7 # define cur_break(a) vlink((a)+1) /*in passive node, points to position of this breakpoint */

2 4

Duplicated italic correction in do_delimiter
by Marcel Fabian Krüger 31 Jan '21

31 Jan '21

Hi, take the following document: \font\extend ntxexx at 10pt \textfont3=\extend \showboxdepth\maxdimen \showboxbreadth\maxdimen \setbox0=\hbox{$ \big) $} \showbox0 \bye It writes to the log: > \box0= \hbox(8.5+3.1125)x4.72997, direction TLT .\mathon .\hbox(8.5+3.1125)x4.72997, direction TLT ..\mathon ..\hbox(8.5+3.1125)x4.72997, direction TLT ...\hbox(0.56+10.665)x4.72997, shifted -7.55249, direction TLT ....\extend ^A ...\vbox(8.5+0.0)x0.0, direction TLT ...\hbox(0.0+0.0)x0.0, direction TLT ..\mathoff .\mathoff Particularly note the width 4.72997 of the hbox with the delimiter. If we look at the relevant part of the tfm file (in pl format), we find the dimensions of the glyph as (CHARACTER O 1 (CHARWD R 0.313) (CHARHT R 0.056) (CHARDP R 1.0665) (CHARIC R 0.08) (NEXTLARGER O 21) ) We would expect the width of the box to be 10pt*(CHARWD+CHARIC)=10pt*(0.313+0.08)=3.93pt. Instead we get 10pt*(CHARWD+2*CHARIC)=10pt*(0.313+2*0.08)=4.73pt~4.72997. The cause of this is in mlist.c, do_delimiter (around line 1652): b = char_box(f, c, att); if (!do_new_math(f)) { /*tex Italic gets added to width. */ width(b) += char_italic(f, c); } The char_box function already adds the italic correction to the box width if new math isn't used, therefore the additionally added char_italic leads to the duplicated value. This can be fixed by removing the if block: diff --git a/source/texk/web2c/luatexdir/tex/mlist.c b/source/texk/web2c/luatexdir/tex/mlist.c index c34dee488..e9df7b13f 100644 --- a/source/texk/web2c/luatexdir/tex/mlist.c +++ b/source/texk/web2c/luatexdir/tex/mlist.c @@ -1649,11 +1649,8 @@ static pointer do_delimiter(pointer q, pointer d, int s, scaled v, boolean flat, if (same != NULL && x == c) { *same = emas; } + /*tex italic gets added to width */ b = char_box(f, c, att); - if (!do_new_math(f)) { - /*tex Italic gets added to width. */ - width(b) += char_italic(f, c); - } if (delta != NULL) { /*tex This used to be (f, x). */ *delta = char_italic(f, c); -- Best regards, Marcel

3 2

LuaTeX: autokenring in math
by Petr Olsak 29 Jan '21

29 Jan '21

Hi LuaTeX developers, please, look at the issue https://github.com/latex3/luaotfload/issues/174 including my comment there. What is your opinion to this? Petr Olsak

1 0

linkarea is wrong if startlink is used in an \rlap
by Ulrike Fischer 13 Jan '21

13 Jan '21

In the following example the link area always starts at the begin of the \rlap (and so includes the yyyy) With pdftex the link area only surrounds the link text as expected X \rlap{yyyyyyy \ifx\pdfextension\undefined \pdfstartlink \else \pdfextension startlink \fi attr {/Border[0 0 1]} user{ /Subtype/Link% /A<<% /Type/Action% /S/URI% /URI(blub)% >>% } Linktext \ifx\pdfextension\undefined \pdfendlink \else \pdfextension endlink \fi } \bye -- Ulrike Fischer http://www.troubleshooting-tex.de/

3 2

keeping in sync with pdftex changes
by Ulrike Fischer 12 Jan '21

12 Jan '21

pdftex will in texlive 2021 change in two places 1. links where \pdfstartlink is in a different boxing level as \pdfendlink are no longer a fatal error but gives only a warning: x\hbox{\pdfstartlink attr {/Border[0 0 1]} user{ /Subtype/Link% /A<<% /Type/Action% /S/URI% /URI(blkub)% >>% } Linktext} \pdfendlink \bye pdfTeX warning: \pdfendlink ended up in different nesting level than \pdfstartl ink 2. There are two new primitives \pdfrunninglinkoff & \pdfrunninglinkon which allow to interrupt a running link for example in the header or footer. Would it be possible to get this changes into luatex too? -- Ulrike Fischer http://www.troubleshooting-tex.de/

4 14

luatex segfault with huge romannumeral overfull
by Hironobu Yamashita 25 Nov '20

25 Nov '20

Hi, I happened to find that luatex and luahbtex crash: $ luatex \\romannumeral2000000000\\end This is LuaTeX, Version 1.13.0 (TeX Live 2021/dev) restricted system commands enabled. Segmentation fault: 11 (I'm using self-build from current TL svn.) Best, Hironobu

2 1

Hyphenation issue regarding frozen glyph nodes
by Marcel Fabian Krüger 09 Nov '20

09 Nov '20

Hi, as recently discussed on the luatex list there is an issue regarding hyphenation ehn a frozen glyph appears next to an explicit hyphenation character. E.g. \setbox0\hbox{X} X-\unhbox0 \bye triggers an assertion error. I noticed that with the latest experimental code a TeX error is triggered instead but IMO this should not be an error. Therefore I suggest to fix this by treating glyph nodes failing is_simple_character just like non-glyph nodes, e.g. with --- source/texk/web2c/luatexdir/lang/texlang.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/texk/web2c/luatexdir/lang/texlang.c b/source/texk/web2c/luatexdir/lang/texlang.c index 48a242fa0..e7635903a 100644 --- a/source/texk/web2c/luatexdir/lang/texlang.c +++ b/source/texk/web2c/luatexdir/lang/texlang.c @@ -1049,7 +1049,7 @@ void hnj_hyphenation(halfword head, halfword tail) if (explicit_hyphen == true) { /*tex we are not at the start, so we only need to look ahead */ halfword t = vlink(r) ; - if ((automatic_hyphen_mode_par == 0 || automatic_hyphen_mode_par == 1) && (t != null) && ((type(t) == glyph_node) && (character(t) != ex_hyphen_char_par))) { + if ((automatic_hyphen_mode_par == 0 || automatic_hyphen_mode_par == 1) && (t != null) && ((type(t) == glyph_node) && is_simple_character(t) && (character(t) != ex_hyphen_char_par))) { /*tex we have a word already but the next character may not be a hyphen too */ r = compound_word_break(r, char_lang(r)); if (compound_hyphen) { @@ -1063,7 +1063,7 @@ void hnj_hyphenation(halfword head, halfword tail) } } else { /*tex we jump over the sequence of hyphens */ - while ((t != null) && (type(t) == glyph_node) && (character(t) == ex_hyphen_char_par)) { + while ((t != null) && (type(t) == glyph_node) && is_simple_character(t) && (character(t) == ex_hyphen_char_par)) { r = t ; t = vlink(r) ; } -- 2.29.2 Best regards, Marcel

2 1