I wrote:
coroutine.wrap(function(get_token) while true local token1 = get_token() if token1.cmd != "^" then get_token = coroutine.yield(token1) else local token2 = get_token() if token2.cmd != "^" then coroutine.yield(token1) get_token = coroutine.yield(token2) else local token3 = get_token() if token3.cmd ... then get_token = coroutine.yield(something) else coroutine.yield(token1) coroutine.yield(token2) get_token = coroutine.yield(token3) end end end end end)
One problem with this approach is that the lookahead kept internally within a coroutine will get lost when one switches the filter function out (not that the current approach fares better here). One solution might be to pass an artificial "EOF" token to the filter function as the last act before removing it from token_filter, and accepting a list of lookahead tokens as the return value.
About that lookahead and phasing the filter routine out: more consistent would probably be the following: when there is no longer input for the filter routine, pass it nil as input routine. Once it bleeds out nil, it is finished. So we get: coroutine.wrap(function(get_token) while get_token != nil local token1 = get_token() if token1.cmd != "^" then get_token = coroutine.yield(token1) else local token2 = get_token() if token2.cmd != "^" then coroutine.yield(token1) get_token = coroutine.yield(token2) else local token3 = get_token() if token3.cmd ... then get_token = coroutine.yield(something) else coroutine.yield(token1) coroutine.yield(token2) get_token = coroutine.yield(token3) end end end end return nil end) Managing multiple filter functions will still be some work, probably requiring the use of a suitable helper function. But basically, I find this sort of interface more natural than the current token_filter semantics. -- David Kastrup