Hi,

according to the documentation for `append_to_vlist_filter`:

> The prevdepth is also optional

so it should be allowed to only return a box in which case the prevdepth
value shouldn't change.

But if we actually try this:

\directlua{
  callback.register("append_to_vlist_filter", function(b)
    return b
  end)
}
abc
\bye

LuaTeX answers with
warning  (pdf backend): no pages of output.

because it discards all boxes. This is caused by a bug in luanode.c, lua_appendtovlist_callback:
The callback is called with lua_pcall(Luas, 4, 2, 0). Then LuaTeX tries
to determine if one or two values have been returned by looking at the
type of slot -1: If this is a number, the second return value have been
a number and it is evaluated as new prevdepth. Otherwise

    else if (lua_type(Luas, -1) != LUA_TNIL) {
        p = check_isnode(Luas, -1);
        *result = *p;
    }

is evaluated to scan for a single return value which is a node. But because
lua_pcall(Luas, 4, 2, 0) explicitly requested two return values, a
callback with one return values would result in a stack where the one
return value is followed by `nil`. Therefore this `if` block is not
entered.

The problem can be fixed  by always looking at stack slot -2 for the
returned node:

 source/texk/web2c/luatexdir/lua/luanode.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/source/texk/web2c/luatexdir/lua/luanode.c b/source/texk/web2c/luatexdir/lua/luanode.c
index a0c97e37d..d4baab96d 100644
--- a/source/texk/web2c/luatexdir/lua/luanode.c
+++ b/source/texk/web2c/luatexdir/lua/luanode.c
@@ -183,12 +183,9 @@ int lua_appendtovlist_callback(halfword box, int location, halfword prev_depth,
     if (lua_type(Luas,-1) == LUA_TNUMBER) {
         *next_depth = lua_roundnumber(Luas,-1);
         *prev_set = true;
-        if (lua_type(Luas, -2) != LUA_TNIL) {
-            p = check_isnode(Luas, -2);
-            *result = *p;
-        }
-    } else if (lua_type(Luas, -1) != LUA_TNIL) {
-        p = check_isnode(Luas, -1);
+    }
+    if (lua_type(Luas, -2) != LUA_TNIL) {
+        p = check_isnode(Luas, -2);
         *result = *p;
     }
     return 1;
--