Taco Hoekwater writes:

Arthur Reutenauer wrote:

...

...

Instead, LuaTeX barfs on "\^^9d" and similar ASCII _transliterations_ of characters which happen to be legal _characters_ in Unicode (though not legal _bytes_ in utf-8).

Good spot, I already noticed there was many problems with latex but I thought it was mainly due to pattern files (and I gave up very early on LaTeX in LuaTeX anyway). I suppose the ^^ notation should yield a UTF-8 encoded sequence and not an individual byte (XeTeX indeed is perfectly happy with it).

It worked before, so I probably messed up something along the way. It is safe to assume there will be a fix in the next snapshot.

Anyway: I think it is a safe assumption that LuaTeX should be able to deal with current versions of LaTeX (I think it would be a mistake to have to rely on lambda). So the kind of utf-8 support (OTP or something) used for Omega needs to be somewhat optional. I don't have any clue about the current implementation, but the amount of error messages I got suggests there are several areas involved. Here is my take on what would constitute a sane environment (some of that probably is already implemented in XeTeX) in my opinion: Single characters: encoded in unicode (UCS-21 or similar). Input line buffer: array of single characters. Characters are created from input by using the input coding system of the file (basically one of 8-bit, utf-8, at some later point of time possibly also things like utf-16-le or utf-16-be). LaTeX would be fixed to "transparent" at first. Which would make it work like before. However, one would want to eventually add something like an utf8l input encoding in order to have it behave more sanely. String space: utf-8 encoded. This is probably incompatible with previous code, but saves space. Log and console output: switchable utf-8 or 8-bit, probably depending on locale and/or inherited from the mode of the current input file. In "8-bit" mode, obviously all characters with a code point above 256 need to get output as ^^^^abcd or ^^^^^^01abcd or similar. Write streams: similar. It might be possible to generally write utf-8, but then it might be a good idea to add a byte order mark at the start of files so that \input on such files will flip the coding system appropriately. I really need to take a look at XeTeX. -- David Kastrup

Taco Hoekwater writes:

David Kastrup wrote:

...

So the kind of utf-8 support (OTP or something) used for Omega needs to be somewhat optional.

No, the error is simply a bug. All I/O characters that are visible to the bare engine is, and will be, utf-8 encoded.

What is "the bare engine"? From the TeX side, one sees Unicode characters.

If you want to do bare bytes, you have to preprocess them in lua.

How do you interpret input bytes that don't form valid utf-8 sequences? As long as they are preserved in some recognizable manner, it should be possible to do this sort of reverse conversion to the original bytes, but it certainly does not sound like it would make for attractive speed. -- David Kastrup

Taco Hoekwater writes:

[...] This is discussed in the reference manual, so if you have not looked at that yet, please do so before replying to this message.

Dangerous advice since this gives me ideas... Here is something I find worth giving a different API: \subsubsection{\callback{token_filter}} This callback allows you to change the fetch and preprocess any lexical token that enters \LUATEX, before \LUATEX\ executes or expands the associated command. \startfunctioncall function () return table <token> end \stopfunctioncall The calling convention for this callback is bit more complicated then for most other callbacks. The function should either return a lua table representing a valid to-be-processed token or tokenlist, or something else like nil or an empty table. If your lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). See the description of \callbacklib{token} for some handy functions to be used in conjunction with this callback. If your function returns a single usable token, then that token will be processed by \LUATEX\ immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack as completely new token list level, with it's token type set to `inserted'. In either case, the returned token(s) will not be fed back into the callback function. I think that I would like to propose a much more luatic solution: If token_filter is set, it is called with one argument \verb|get_next|, the function originally supposed to get the next token. token_filter should then call this function as often as it needs to (possibly zero times) and return one token to the caller. If you need to readahead and buffer tokens (like when simulating OTPs), the easiest way to do this is using something like the following for the filter function: coroutine.wrap(function(get_token) while true local token1 = get_token() if token1.cmd != "^" then get_token = coroutine.yield(token1) else local token2 = get_token() if token2.cmd != "^" then coroutine.yield(token1) get_token = coroutine.yield(token2) else local token3 = get_token() if token3.cmd ... then get_token = coroutine.yield(something) else coroutine.yield(token1) coroutine.yield(token2) get_token = coroutine.yield(token3) end end end end end) Ok, the code itself is nonsensically, but it should illustrate the working principle: if the filtering is not 1:1, one can use a coroutine for analysing the input, buffering and producing the tokens. This approach also has the advantage that one can stack filter functions easily. The existing interface makes that much harder: I actually have no good idea how one would go about it. One problem with this approach is that the lookahead kept internally within a coroutine will get lost when one switches the filter function out (not that the current approach fares better here). One solution might be to pass an artificial "EOF" token to the filter function as the last act before removing it from token_filter, and accepting a list of lookahead tokens as the return value. -- David Kastrup

David@lola.quinscape.zz wrote:

Taco Hoekwater writes:

...

[...] This is discussed in the reference manual, so if you have not looked at that yet, please do so before replying to this message.

Dangerous advice since this gives me ideas...

Here is something I find worth giving a different API:

\subsubsection{\callback{token_filter}}

This callback allows you to change the fetch and preprocess any lexical token that enters \LUATEX, before \LUATEX\ executes or expands the associated command.

\startfunctioncall function () return table <token> end \stopfunctioncall

The calling convention for this callback is bit more complicated then for most other callbacks. The function should either return a lua table representing a valid to-be-processed token or tokenlist, or something else like nil or an empty table.

If your lua function does not return a table representing a valid token, it will be immediately called again, until it eventually does return a useful token or tokenlist (or until you reset the callback value to nil). See the description of \callbacklib{token} for some handy functions to be used in conjunction with this callback.

If your function returns a single usable token, then that token will be processed by \LUATEX\ immediately. If the function returns a token list (a table consisting of a list of consecutive token tables), then that list will be pushed to the input stack as completely new token list level, with it's token type set to `inserted'. In either case, the returned token(s) will not be fed back into the callback function.

I think that I would like to propose a much more luatic solution:

If token_filter is set, it is called with one argument \verb|get_next|, the function originally supposed to get the next token.

token_filter should then call this function as often as it needs to (possibly zero times) and return one token to the caller.

If you need to readahead and buffer tokens (like when simulating OTPs), the easiest way to do this is using something like the following for the filter function:

coroutine.wrap(function(get_token) while true local token1 = get_token() if token1.cmd != "^" then get_token = coroutine.yield(token1) else local token2 = get_token() if token2.cmd != "^" then coroutine.yield(token1) get_token = coroutine.yield(token2) else local token3 = get_token() if token3.cmd ... then get_token = coroutine.yield(something) else coroutine.yield(token1) coroutine.yield(token2) get_token = coroutine.yield(token3) end end end end end)

Ok, the code itself is nonsensically, but it should illustrate the working principle: if the filtering is not 1:1, one can use a coroutine for analysing the input, buffering and producing the tokens. This approach also has the advantage that one can stack filter functions easily.

The existing interface makes that much harder: I actually have no good idea how one would go about it.

One problem with this approach is that the lookahead kept internally within a coroutine will get lost when one switches the filter function out (not that the current approach fares better here). One solution might be to pass an artificial "EOF" token to the filter function as the last act before removing it from token_filter, and accepting a list of lookahead tokens as the return value.

the problem is that this is real slow which renders it rather unusable, even the current implementation is already on the edge of acceptable why do you want to handle the ^'s? you can do that using the input line callback Hans -- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Hans Hagen wrote:

...

If you need to readahead and buffer tokens (like when simulating OTPs), the easiest way to do this is using something like the following for the filter function:

If you need to read ahead for tokens, just run token.get_next() in a loop that stores tokens in a local table until you are happy. Then return that table after processing it. There is no need to return to the TeX control loop before that. Best, Taco

David Kastrup wrote:

Taco Hoekwater writes:

...

David Kastrup wrote:

...

So the kind of utf-8 support (OTP or something) used for Omega needs to be somewhat optional.

No, the error is simply a bug. All I/O characters that are visible to the bare engine is, and will be, utf-8 encoded.

What is "the bare engine"? From the TeX side, one sees Unicode characters.

...

If you want to do bare bytes, you have to preprocess them in lua.

How do you interpret input bytes that don't form valid utf-8 sequences? As long as they are preserved in some recognizable manner, it should be possible to do this sort of reverse conversion to the original bytes, but it certainly does not sound like it would make for attractive speed.

you can define a callback that will intercept each line and do whatever you want with the content as long as what you pipe back into tex is utf 8 the internal dataflow is utf8 and as the manual states, getting not utf (8 bit) out is a matter of remapping to a reserved private area in unicode (for instance, pdf literals may need 8 bit instead of utf, and that's how it's done) this keeps luatex internally clean, but permits macro writers to do what they want; it's also the principle of luatex ... provide access and points of interception but stay as clean as possible internally anyhow, good old tex was never 8 bit clean (at least not till recently and then only with natural.tcx or -8bit) also keep in mind that macro packages need to adapt to luatex and not the reverse -) Hans -- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------