Hello, (This is not strictly LuaTeX-related, though LuaTeX exhibits the same behaviour. But the LuaTeX ML is the only TeX-related forum I currently have access to. And maybe this is something to fix in LuaTeX ;-) While writing unit tests for some of my macros, I stumbled on the following weird behaviour when expanding \toks registers inside \edef using \the (PlainTeX): ---------------------------------- CUT ------------------------------- \catcode`\@=11 \def\log#1{\immediate\write\sixt@@n{#1}} \toks@{|##|} % Obvious result: Doubled \edef\foo{\the\toks@} \log{1: \meaning\foo} % Obvious as well: Doubled \edef\foo{\the\toks@\the\toks@} \log{2: \meaning\foo} % Now this is weird: First doubled, then not \edef\foo{\the\expandafter\toks@\the\toks@} \log{3: \meaning\foo} \bye ---------------------------------- CUT ------------------------------- This produces the following output: ---------------------------------- CUT ------------------------------- TeX: $ tex ToksSharpTest.tex This is TeX, Version 3.141592 (Web2C 7.5.5) (./ToksSharpTest.tex 1: macro:->|####| 2: macro:->|####||####| 3: macro:->|####||##| ) No pages of output. Transcript written on ToksSharpTest.log. pdfTeX: $ pdftex ToksSharpTest.tex This is pdfTeX, Version 3.141592-1.40.3 (Web2C 7.5.6) entering extended mode (./ToksSharpTest.tex 1: macro:->|####| 2: macro:->|####||####| 3: macro:->|####||##| ) No pages of output. Transcript written on ToksSharpTest.log. LuaTeX: $ luatex ToksSharpTest.tex This is luaTeX, Version 3.141592-beta-0.11.2-2007091918 (Web2C 7.5.6) (ToksSharpTest.tex 1: macro:->|####| 2: macro:->|####||####| 3: macro:->|####||##| ) No pages of output. Transcript written on ToksSharpTest.log. ---------------------------------- CUT ------------------------------- So when jumping out of the first expansion of the \toks register and into the second expansion, TeX seems to forget that it is inside an \edef and thus does not double the ##. I do not know what to make of this. I could not find any place in the TeXbook where it is stated that # is doubled when expanding a \toks register inside \edef, so this might not be a bug, but simply some weird undocumented phenomenon. And it might be useful too, to build up the body of a complicated macro inside a \toks register before finally creating the macro using \edef: % Empty dummy register \toksA{} % Macro body (not really complicated) \toksB{[#1] [#2]} % Define the macro \edef\macro#1#2{\the\expandafter\toksA\the\toksB} \meaning\macro would then be: macro:#1#2->[#1] [#2] What do you think? Jonathan
Jonathan Sauer wrote:
Hello,
(This is not strictly LuaTeX-related, though LuaTeX exhibits the same behaviour. But the LuaTeX ML is the only TeX-related forum I currently have access to. And maybe this is something to fix in LuaTeX ;-)
While writing unit tests for some of my macros, I stumbled on the following weird behaviour when expanding \toks registers inside \edef using \the (PlainTeX):
weird indeed, it is some lookahead interference, if you put a relax in between (6x), or do a one level expansion (9x) .... \starttext \newtoks\mytoks \def\log{\immediate\write16} \let\ea\expandafter \toks 0{[##]} \mytoks{[##]} \edef\foo{\the\toks0} \log{1a: \meaning\foo} \edef\foo{\the\toks0\the\toks0} \log{2a: \meaning\foo} \edef\foo{\the\toks0 \the\toks0} \log{3a: \meaning\foo} \edef\foo{\the\toks0 \the\toks0} \log{3a: \meaning\foo} \edef\foo{\the\ea\toks\ea0\the\toks0} \log{4a: \meaning\foo} \edef\foo{\the\ea\toks\ea0\ea\space\the\toks0} \log{5a: \meaning\foo} \edef\foo{\the\mytoks} \log{1b: \meaning\foo} \edef\foo{\the\mytoks\the\mytoks} \log{3b: \meaning\foo} \edef\foo{\the\ea\mytoks\the\mytoks} \log{4b: \meaning\foo} \edef\foo{\the\mytoks\relax\the\mytoks} \log{6x: \meaning\foo} \edef\foo{\the\mytoks[##]} \log{7x: \meaning\foo} \edef\foo{[##][##]} \log{8x: \meaning\foo} \ea\edef\ea\foo\ea{\the\mytoks\the\mytoks} \log{9x: \meaning\foo} \stoptext this # duplication has always been kind of messy (changing catcodes also can have unexpected results because the # is kind of hardcoded in some places) and changing this behaviour may break existing code that happens to work (i have some places where trial and error and tracing finally gave me some idea to handle it in certain cases) (i can imagine a future version of luatex to support an alternative mechanism for arguments, but that will be a few years away) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
"Jonathan Sauer"
While writing unit tests for some of my macros, I stumbled on the following weird behaviour when expanding \toks registers inside \edef using \the (PlainTeX):
There is nothing strange here.
\catcode`\@=11 \def\log#1{\immediate\write\sixt@@n{#1}}
\toks@{|##|}
% Obvious result: Doubled \edef\foo{\the\toks@}
\log{1: \meaning\foo}
% Obvious as well: Doubled \edef\foo{\the\toks@\the\toks@}
\log{2: \meaning\foo}
% Now this is weird: First doubled, then not \edef\foo{\the\expandafter\toks@\the\toks@}
\the\toks@ passes # unmolested into \edef. In every other case (apart from the equivalent use of \unexpanded), a doubled ## is required for producing a single # in the result, because then # is a parameter delimiting token for the \edef. The \expandafter takes the ## out of the token register and into the regular input stream, so the #-preserving character of \the inside of \edef is no longer in action.
So when jumping out of the first expansion of the \toks register and into the second expansion, TeX seems to forget that it is inside an \edef and thus does not double the ##.
TeX _never_ doubles the # in \edef: you are confusing this with a printing artifact.
I do not know what to make of this. I could not find any place in the TeXbook where it is stated that # is doubled when expanding a \toks register inside \edef, so this might not be a bug, but simply some weird undocumented phenomenon.
It is not doubled, but simply left uninterpreted.
And it might be useful too, to build up the body of a complicated macro inside a \toks register before finally creating the macro using \edef:
What do you think?
You are about 20 years late. Please check LaTeX's definition (and usage) of \g@addto@macro: \long\def\g@addto@macro#1#2{% \begingroup \toks@\expandafter{#1#2}% \xdef#1{\the\toks@}% \endgroup} \def\AtEndOfPackage{% \expandafter\g@addto@macro\csname\@currname.\@currext-h@@k\endcsname} \let\AtEndOfClass\AtEndOfPackage \@onlypreamble\AtEndOfPackage \@onlypreamble\AtEndOfClass \def\AtBeginDocument{\g@addto@macro\@begindocumenthook} \def\AtEndDocument{\g@addto@macro\@enddocumenthook} -- David Kastrup
Hello,
[...]
% Now this is weird: First doubled, then not \edef\foo{\the\expandafter\toks@\the\toks@}
\the\toks@ passes # unmolested into \edef. In every other case (apart from the equivalent use of \unexpanded), a doubled ## is required for producing a single # in the result, because then # is a parameter delimiting token for the \edef. The \expandafter takes the ## out of the token register and into the regular input stream, so the #-preserving character of \the inside of \edef is no longer in action.
You are of course right. And I am a little bit wiser.
[...]
And it might be useful too, to build up the body of a complicated macro inside a \toks register before finally creating the macro using \edef:
What do you think?
You are about 20 years late. Please check LaTeX's definition (and usage) of \g@addto@macro: [...]
Well, my example was to show how to use above behaviour to create the body of a macro *with parameters* in a \toks register. \g@addto@macro does not do this.
David Kastrup
Jonathan
"Jonathan Sauer"
[...]
% Now this is weird: First doubled, then not \edef\foo{\the\expandafter\toks@\the\toks@}
\the\toks@ passes # unmolested into \edef. In every other case (apart from the equivalent use of \unexpanded), a doubled ## is required for producing a single # in the result, because then # is a parameter delimiting token for the \edef. The \expandafter takes the ## out of the token register and into the regular input stream, so the #-preserving character of \the inside of \edef is no longer in action.
You are of course right. And I am a little bit wiser.
[...]
And it might be useful too, to build up the body of a complicated macro inside a \toks register before finally creating the macro using \edef:
What do you think?
You are about 20 years late. Please check LaTeX's definition (and usage) of \g@addto@macro: [...]
Well, my example was to show how to use above behaviour to create the body of a macro *with parameters* in a \toks register. \g@addto@macro does not do this.
But it can be used for it. Which is the reason it is implemented using a token register (according to the change note in ltclass.dtx, this has been done in 1994, probably due to a somewhat pressing suggestion of mine, so I have to take back the "20 years"). -- David Kastrup
participants (3)
-
David Kastrup -
Hans Hagen -
Jonathan Sauer