Jonathan Sauer wrote:
Hello,
- When setting a \toks register in Lua using tex.toks['foo'] = 'bar' et.al., what catcodetable is used? From looking inside luatex.web, I would guess that, since str_toks is used, all characters get catcode 12, except for spaces, which get catcode 10. Correct. It is one of the first things I wrote, and it definately needs updating. I now believe the toks array should accept and return token list tables (instead of lua strings), and there should be helper functions to_string() and to_tokenlist() for going back and forth. That needs a bit of (not-yet-done) programming, but it is not hard at all.
I'm wondering how this would affect performance. How costly is the conversion from a string to a table?
costly; input -> texscanner -> internal tex structure with callback or toks or ...: input -> texscanner -> callback that gets token being table -> returns table (or not) -> internal tex structure so, in the tokentable case there are two conversions (+ allocation and garbage collection for the token table) which is way slower
And what about tex.print? Since it creates/writes tokens as well, will it be changed to accept token list tables, too? (at any rate, I think both should accept the same parameter type)
no, that will be string based; pushing strings into texs scanner is currently pretty efficient and te need to convert them to tokens first would make the interface clumsy; so tex.tprint (token print) would be a natural candidate then (one argument, a table of tokens) in practice tex.sprint is used more often than pushing token tables into tex Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------