[Dev-luatex] # or ##, that is the question

[Hans Hagen]

Jonathan Sauer wrote:
> Hello,
> 
> (This is not strictly LuaTeX-related, though LuaTeX exhibits the same
> behaviour. But the LuaTeX ML is the only TeX-related forum I currently
> have access to. And maybe this is something to fix in LuaTeX ;-)
> 
> While writing unit tests for some of my macros, I stumbled on the 
> following weird behaviour when expanding \toks registers inside \edef 
> using \the (PlainTeX):

weird indeed, it is some lookahead interference, if you put a relax in 
between (6x), or do a one level expansion (9x) ....

\starttext

\newtoks\mytoks
\def\log{\immediate\write16}
\let\ea\expandafter

\toks 0{[##]}
\mytoks{[##]}

\edef\foo{\the\toks0}                          \log{1a: \meaning\foo}
\edef\foo{\the\toks0\the\toks0}                \log{2a: \meaning\foo}
\edef\foo{\the\toks0 \the\toks0}               \log{3a: \meaning\foo}
\edef\foo{\the\toks0 \the\toks0}               \log{3a: \meaning\foo}
\edef\foo{\the\ea\toks\ea0\the\toks0}          \log{4a: \meaning\foo}
\edef\foo{\the\ea\toks\ea0\ea\space\the\toks0} \log{5a: \meaning\foo}

\edef\foo{\the\mytoks}                         \log{1b: \meaning\foo}
\edef\foo{\the\mytoks\the\mytoks}              \log{3b: \meaning\foo}
\edef\foo{\the\ea\mytoks\the\mytoks}           \log{4b: \meaning\foo}

\edef\foo{\the\mytoks\relax\the\mytoks}        \log{6x: \meaning\foo}
\edef\foo{\the\mytoks[##]}                     \log{7x: \meaning\foo}
\edef\foo{[##][##]}                            \log{8x: \meaning\foo}
\ea\edef\ea\foo\ea{\the\mytoks\the\mytoks}     \log{9x: \meaning\foo}

\stoptext

this # duplication has always been kind of messy (changing catcodes also 
can have unexpected results because the # is kind of hardcoded in some 
places) and changing this behaviour may break existing code that happens 
to work (i have some places where trial and error and tracing finally 
gave me some idea to handle it in certain cases)

(i can imagine a future version of luatex to support an alternative 
mechanism for arguments, but that will be a few years away)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------