Hi,

current versions of LuaTeX from TeXLive for Linux (Windows is not affected) always
add `\adjdemerits` when a line is more loose than the previous one, even
if it is only one step more loose:

Take

\tracingparagraphs1
\tracingonline1

aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa
aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa
aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa
\bye

This gives

@firstpass
[]\tenrm aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa
aaaaaa aaaaaa aaaaaa
@ via @0 b=96 p=0 d=1011236
@@1: line 1.1 t=21236 -> @0
aaaaaa
@ via @0 b=83 p=0 d=8649
@@2: line 1.3 t=8649 -> @0
aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa aa
aaaa aaaaaa
@ via @1 b=3 p=0 d=169
@@3: line 2.2 t=1011405 -> @1
aaaaaa
@ via @2 b=3 p=0 d=1000169
@@4: line 2.2 t=1008818 -> @2
aaaaaa aaaaaa aaaaaa aaaaaa aaaaaa
@\par via @3 b=0 p=-10000 d=100
@\par via @4 b=0 p=-10000 d=100
@@5: line 3.2- t=1008918 -> @4

Here, the relevant part is

@@2: line 1.3 t=8649 -> @0
...
@ via @2 b=3 p=0 d=1000169
@@4: line 2.2 t=1008818 -> @2

So breakpoint @2 is .3 aka. "tight", while @4 is .2 aka. "decent".
This break gives demerits d=100169=1000000+(10+3)^2. So \adjdemerits are
inserted which is wrong because tight and decent lines are "visually
compatible" as defined in the TeXbook.

The problem is in linebreak.c. There the demerits are inserted by

    if (abs(fit_class - fitness(r)) > 1)
        d = d + adj_demerits;

Both fit_class and fitness(r) have type fitness_value which is an enum
with four entries. These enums have an implementation defined type and
the bug is triggered if LuaTeX is compiled with a compiler which uses an
unsigned type: Then `fit_class - fitness(r)` is a difference of unsigned
values and therefore unsigned. If fitness(r) is bigger than fit_class,
the attempt to store a negative difference in a unsigned value
wraps around to a high positive value which is bigger than 1. So if
fitness(r) is bigger than fit_class, the condition is always true.

This can be fixed by explicitly casting fit_class and fitness(r) to a
signed type. A possible patch is attached.