# [NTG-context] waiting for mswincontext / injecting

Hans Hagen pragma at wxs.nl
Sat Sep 22 12:34:10 CEST 2007

Wolfgang Schuster wrote:

> Version 0.997 of XeTeX introduced a new concept to assign every
> character a class value. You can define rules what should happen if you
> switch for example from class 1 to class 2 and another one if you do
> the same in reverse order. The commands for this action are saved in
> token register and came into action if this class switch appear in the
> document.

ok, i remember jonathan talking about it at bachotek

> the unicode-letters file from XeTeX define a class for quotation,
> punctutation ... in CJK and another one for the other CJK characters
> and define spacing and linebreaking between normal characters and
> punctuation now by the class number (you did something similiar in your
> chinese module but used active characters instead).

the problem with such mechanisms is limited options for lookahead and
look back

> The XeTeX reference manual has a simple example of this concept on page
> 10 and 11.
>
> http://tug.ctan.org/info/xetexref/XeTeX-reference.pdf
>
> can't  this be done in LuaTeX with nodes (or what you mentioned for
> your color modell) or something similar.

injecting tokens into the nodelist is no option, since at that stage
tokenization already has been done; so it should happen earlier, during
tokenization, which can be kind of messy, because even then we're
talking of

a\beta c -> a\char[beta]c (no further expansion) -> inject expanded
stuff before, after each chartoken or \char token -> convert tokens to
nodes

sure, i can implement such an injector (since it's tokens, it should
haven in the input stream somehow, operating on node lists is kind of
messy in that case because tokens need to be expanded) but we're not
going to use that method in mkiv. It's one of those areas where xetex
and luatex support differs a lot.

> but I am too lazy to rename files or create them always by hand with
> every update ...

that's what scripting is fore, after all, format generation in tex
engines is not automated at all (one has to manage moving fmt files to
the right place anyway)

Hans

-----------------------------------------------------------------