On Wed, Oct 30, 2019 at 2:33 AM Marcel Fabian Krüger
Hi,
current versions of LuaTeX from TeXLive for Linux (Windows is not affected) always add `\adjdemerits` when a line is more loose than the previous one, even if it is only one step more loose:
Take
\tracingparagraphs1 \tracingonline1
aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa \bye
This gives
@firstpass []\tenrm aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa @ via @0 b=96 p=0 d=1011236 @@1: line 1.1 t=21236 -> @0 aaaaaa @ via @0 b=83 p=0 d=8649 @@2: line 1.3 t=8649 -> @0 aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aa aaaa aaaaaa @ via @1 b=3 p=0 d=169 @@3: line 2.2 t=1011405 -> @1 aaaaaa @ via @2 b=3 p=0 d=1000169 @@4: line 2.2 t=1008818 -> @2 aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa @\par via @3 b=0 p=-10000 d=100 @\par via @4 b=0 p=-10000 d=100 @@5: line 3.2- t=1008918 -> @4
Here, the relevant part is
@@2: line 1.3 t=8649 -> @0 ... @ via @2 b=3 p=0 d=1000169 @@4: line 2.2 t=1008818 -> @2
So breakpoint @2 is .3 aka. "tight", while @4 is .2 aka. "decent". This break gives demerits d=100169=1000000+(10+3)^2. So \adjdemerits are inserted which is wrong because tight and decent lines are "visually compatible" as defined in the TeXbook.
The problem is in linebreak.c. There the demerits are inserted by
if (abs(fit_class - fitness(r)) > 1) d = d + adj_demerits;
Both fit_class and fitness(r) have type fitness_value which is an enum with four entries. These enums have an implementation defined type and the bug is triggered if LuaTeX is compiled with a compiler which uses an unsigned type: Then `fit_class - fitness(r)` is a difference of unsigned values and therefore unsigned. If fitness(r) is bigger than fit_class, the attempt to store a negative difference in a unsigned value wraps around to a high positive value which is bigger than 1. So if fitness(r) is bigger than fit_class, the condition is always true.
This can be fixed by explicitly casting fit_class and fitness(r) to a signed type. A possible patch is attached.
hm. I will see it. -- luigi