On Thu, Jun 27, 2019 at 2:01 PM Marcel Fabian Krüger <tex@2krueger.de> wrote:

Hi everyone,

in current LuaTeX versions, the result of `linebreak` filter is ignored
and instead it is always treated as if no node has been returned (so the
internal linebreak algorithm is applied). A minimal example:


\directlua{
callback.register("linebreak_filter", function(nodes)
  return node.hpack(nodes)
end)
}
Some very very very very very very very very very very very very very
very very very very very very very very very very very very very very
very very very very very very very very very very very very very very
very very very very very very very very very very very very very very
very very very very very very very very very very very very very very
very very very long paragraph.
\bye

You would expect the whole paragraph to be set in a single line, but
normal paragraph breaking is applies instead. This is caused by a change
in revision 6686 from Mar 29 2018 "experimental) possible fix for for lua
stack issue + consistent error handling (HH)". There in the function
lua_linebreak_callback, the change

         return ret;
     }
+    lua_settop(Luas, s_top);
     p = lua_touserdata(Luas, -1);
     if (p != NULL) {
         a = nodelist_from_lua(Luas);
         try_couple_nodes(*new_head,a);
         ret = 1;
     }
-    lua_settop(Luas, s_top);
     return ret;

has been applied. I am not sure what kind of stack issue this was
supposed to fix, but as a side effect this moves the check for the node
return value after `lua_settop`, so the return value is already removed
from the stack when the test is done. So it is ignored.

hm, I will try reverting the code to see when the stack issue appears.