[Dev-luatex] # or ##, that is the question

17 Oct 2007

      Hello,

(This is not strictly LuaTeX-related, though LuaTeX exhibits the same
behaviour. But the LuaTeX ML is the only TeX-related forum I currently
have access to. And maybe this is something to fix in LuaTeX ;-)

While writing unit tests for some of my macros, I stumbled on the 
following weird behaviour when expanding \toks registers inside \edef 
using \the (PlainTeX):

---------------------------------- CUT -------------------------------

\catcode`\@=11
\def\log#1{\immediate\write\sixt@@n{#1}}

\toks@{|##|}

% Obvious result: Doubled
\edef\foo{\the\toks@}

\log{1: \meaning\foo}

% Obvious as well: Doubled
\edef\foo{\the\toks@\the\toks@}

\log{2: \meaning\foo}

% Now this is weird: First doubled, then not
\edef\foo{\the\expandafter\toks@\the\toks@}

\log{3: \meaning\foo}

\bye

---------------------------------- CUT -------------------------------

This produces the following output:

---------------------------------- CUT -------------------------------

TeX:

$ tex ToksSharpTest.tex 
This is TeX, Version 3.141592 (Web2C 7.5.5)
(./ToksSharpTest.tex
1: macro:->|####|
2: macro:->|####||####|
3: macro:->|####||##|
 )
No pages of output.
Transcript written on ToksSharpTest.log.

pdfTeX:

$ pdftex ToksSharpTest.tex 
This is pdfTeX, Version 3.141592-1.40.3 (Web2C 7.5.6)
entering extended mode
(./ToksSharpTest.tex
1: macro:->|####|
2: macro:->|####||####|
3: macro:->|####||##|
 )
No pages of output.
Transcript written on ToksSharpTest.log.

LuaTeX:

$ luatex ToksSharpTest.tex 
This is luaTeX, Version 3.141592-beta-0.11.2-2007091918 (Web2C 7.5.6)
(ToksSharpTest.tex
1: macro:->|####|
2: macro:->|####||####|
3: macro:->|####||##|
 )
No pages of output.
Transcript written on ToksSharpTest.log.

---------------------------------- CUT -------------------------------

So when jumping out of the first expansion of the \toks register and
into the second expansion, TeX seems to forget that it is inside an 
\edef and thus does not double the ##.

I do not know what to make of this. I could not find any place in the
TeXbook where it is stated that # is doubled when expanding a \toks
register inside \edef, so this might not be a bug, but simply some 
weird undocumented phenomenon.

And it might be useful too, to build up the body of a complicated macro
inside a \toks register before finally creating the macro using \edef:

% Empty dummy register
\toksA{}

% Macro body (not really complicated)
\toksB{[#1] [#2]}

% Define the macro
\edef\macro#1#2{\the\expandafter\toksA\the\toksB}

\meaning\macro would then be: macro:#1#2->[#1] [#2]

What do you think?

Jonathan

[Dev-luatex] # or ##, that is the question

Jonathan Sauer