Assignments in runtoks
Hello, in current versions of LuaTeX, `runtoks` does not preserve token lists of ongoing assignments. Therefore running `runtoks` inside of `\edef` or similar replace the definition with the defined value of the last inner assignment. For example: \toks0{\def\innermacro{inner}} \edef\outermacro{outer\directlua{tex.runtoks(0)}} \show\outermacro \bye in plain TeX should show
\outermacro=macro: ->outer.
but currently does show
\outermacro=macro: ->inner.
instead. (As a side effect the reference count isn't updated, so for example \toks0{\def\innermacro{inner}\let\innermacro\undefined} \edef\outermacro{outer\directlua{tex.runtoks(0)}} \show\outermacro \bye starts printing hyphenation patterns on my system.) This can be fixed by saving scanner_status, warning_index and def_ref when entering local_control: diff --git a/source/texk/web2c/luatexdir/tex/maincontrol.c b/source/texk/web2c/luatexdir/tex/maincontrol.c index b01931dd1..284fb0ae3 100644 --- a/source/texk/web2c/luatexdir/tex/maincontrol.c +++ b/source/texk/web2c/luatexdir/tex/maincontrol.c @@ -1055,6 +1055,10 @@ extern void local_control_message(const char *s) void local_control(void) { int ll = local_level; + int save_scanner_status = scanner_status; + halfword save_def_ref = def_ref; + halfword save_warning_index = warning_index; + main_control_state = goto_next; local_level += 1; while (1) { @@ -1077,14 +1081,17 @@ void local_control(void) if (tracing_nesting_par > 2) { local_control_message("leaving due to level change"); } - return ; + break; } else if (main_control_state == goto_return) { if (tracing_nesting_par > 2) { local_control_message("leaving due to triggering"); } - return; + break; } } + scanner_status = save_scanner_status; + def_ref = save_def_ref; + warning_index = save_warning_index; return; }
On Thu, Aug 29, 2019 at 2:42 PM Marcel Fabian Krüger
Hello,
in current versions of LuaTeX, `runtoks` does not preserve token lists of ongoing assignments. Therefore running `runtoks` inside of `\edef` or similar replace the definition with the defined value of the last inner assignment.
For example:
\toks0{\def\innermacro{inner}} \edef\outermacro{outer\directlua{tex.runtoks(0)}} \show\outermacro \bye
in plain TeX should show
\outermacro=macro: ->outer.
but currently does show
\outermacro=macro: ->inner.
instead.
(As a side effect the reference count isn't updated, so for example
\toks0{\def\innermacro{inner}\let\innermacro\undefined} \edef\outermacro{outer\directlua{tex.runtoks(0)}} \show\outermacro \bye
starts printing hyphenation patterns on my system.)
This can be fixed by saving scanner_status, warning_index and def_ref when entering local_control:
diff --git a/source/texk/web2c/luatexdir/tex/maincontrol.c b/source/texk/web2c/luatexdir/tex/maincontrol.c index b01931dd1..284fb0ae3 100644 --- a/source/texk/web2c/luatexdir/tex/maincontrol.c +++ b/source/texk/web2c/luatexdir/tex/maincontrol.c @@ -1055,6 +1055,10 @@ extern void local_control_message(const char *s) void local_control(void) { int ll = local_level; + int save_scanner_status = scanner_status; + halfword save_def_ref = def_ref; + halfword save_warning_index = warning_index; + main_control_state = goto_next; local_level += 1; while (1) { @@ -1077,14 +1081,17 @@ void local_control(void) if (tracing_nesting_par > 2) { local_control_message("leaving due to level change"); } - return ; + break; } else if (main_control_state == goto_return) { if (tracing_nesting_par > 2) { local_control_message("leaving due to triggering"); } - return; + break; } } + scanner_status = save_scanner_status; + def_ref = save_def_ref; + warning_index = save_warning_index; return; }
we are seeing it. -- luigi
On 8/29/2019 2:42 PM, Marcel Fabian Krüger wrote:
Hello,
in current versions of LuaTeX, `runtoks` does not preserve token lists of ongoing assignments. Therefore running `runtoks` inside of `\edef` or similar replace the definition with the defined value of the last inner assignment.
fixed in next luatex but in a different way as we then also need to support deeper nesting \toks2{\def\deepermacro{deeper}} \toks0{\def\innermacro{inner}\def\deepermacro{weird}\directlua{tex.runtoks(2)}} \edef\outermacro{outer\directlua{tex.runtoks(0)}} \meaning\innermacro \par \meaning\deepermacro\par \meaning\outermacro \par ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
On Sun, Sep 1, 2019 at 2:28 PM Hans Hagen
On 8/29/2019 2:42 PM, Marcel Fabian Krüger wrote:
Hello,
in current versions of LuaTeX, `runtoks` does not preserve token lists of ongoing assignments. Therefore running `runtoks` inside of `\edef` or similar replace the definition with the defined value of the last inner assignment.
fixed in next luatex but in a different way as we then also need to support deeper nesting
\toks2{\def\deepermacro{deeper}}
\toks0{\def\innermacro{inner}\def\deepermacro{weird}\directlua{tex.runtoks(2)}}
\edef\outermacro{outer\directlua{tex.runtoks(0)}}
\meaning\innermacro \par \meaning\deepermacro\par \meaning\outermacro \par
done ------------------------------------------------------------------------ r7186 | luigi.scarso@gmail.com | 2019-09-01 15:25:05 +0200 (Sun, 01 Sep 2019) | 1 line permit nested runtoks in more complex situations of nesting (HH) ------------------------------------------------------------------------ -- luigi
fixed in next luatex but in a different way as we then also need to support deeper nesting
\toks2{\def\deepermacro{deeper}}
\toks0{\def\innermacro{inner}\def\deepermacro{weird}\directlua{tex.runtoks(2)}}
\edef\outermacro{outer\directlua{tex.runtoks(0)}}
\meaning\innermacro \par \meaning\deepermacro\par \meaning\outermacro \par
done ------------------------------------------------------------------------ r7186 | luigi.scarso@gmail.com | 2019-09-01 15:25:05 +0200 (Sun, 01 Sep 2019) | 1 line
permit nested runtoks in more complex situations of nesting (HH) ------------------------------------------------------------------------
Awesome, thank you. One further question out of academic interest: What was the problem with my former proposed fix? It seems to give the right result in your example and I currently can't see why it would have problems with nesting. Marcel
fixed in next luatex but in a different way as we then also need to support deeper nesting
\toks2{\def\deepermacro{deeper}}
\toks0{\def\innermacro{inner}\def\deepermacro{weird}\directlua{tex.runtoks(2)}}
\edef\outermacro{outer\directlua{tex.runtoks(0)}}
\meaning\innermacro \par \meaning\deepermacro\par \meaning\outermacro \par
done ------------------------------------------------------------------------ r7186 | luigi.scarso@gmail.com | 2019-09-01 15:25:05 +0200 (Sun, 01 Sep 2019) | 1 line
permit nested runtoks in more complex situations of nesting (HH) ------------------------------------------------------------------------
Awesome, thank you. One further question out of academic interest: What was the problem with my former proposed fix? It seems to give the right result in your example and I currently can't see why it would have problems with nesting. It's more a matter of symmetry ... to have it in the end_local handler so that it happens at the same level. (Ok, I could change that but first I need to ponder other side effects like node being produced when in an edef. I knew that was an issue and could live with that because of chicken-egg issues, but still. Maybe some variants in due time. Low
On 9/1/2019 3:39 PM, Marcel Fabian Krüger wrote: priority stuff.) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------
participants (3)
-
Hans Hagen -
luigi scarso -
Marcel Fabian Krüger