Am 03.07.2014 um 12:49 schrieb Sanjoy Mahajan :

I thought the problem might lie with the general \definemathunderextensible (math-stc.mkvi), because underbrace is defined as

\definemathunderextensible [vfenced] [underbrace] ["FE3DF] % ["023DF]

Other underextensibles include \underbar:

\definemathunderextensible [vfenced] [underbar] ["FE33F] % ["203E]

So I tried my example replacing \underbrace with \underbar but the alignment comes out fine (and \overbrace works fine too). So that debugging idea didn't work, except to indicate that something special about underbrace isn't working right.

Does anyone have suggestions for further debugging ideas?

math-stc.mkvi: \unexpanded\def\math_stackers_make_double#top#bottom#category#codepoint#codeextra#text% {\begingroup \edef\currentmathstackers{#category}% \mathstackersparameter\c!left\relax \ifmmode\math_class_by_parameter\mathstackersparameter\else\dontleavehmode\fi {\edef\currentmathstackers{#category}% \edef\m_math_stackers_text_middle {#text}% % ... % - \ifx\currentmathtext\empty + \ifx\m_math_stackers_text_middle\empty \setbox\scratchboxthree\emptyhbox \else \setmathtextbox\scratchboxthree\hbox{\math_stackers_middletext}% \fi ... \fi}% \mathstackersparameter\c!right\relax \edef\p_limits{\mathstackersparameter\c!mathlimits}% \ifx\p_limits\v!yes \expandafter\endgroup\expandafter\limits \else \expandafter\endgroup \fi} Wolfgang

On 7/3/2014 11:03 PM, Sanjoy Mahajan wrote:

...

math-stc.mkvi:

\unexpanded\def\math_stackers_make_double#top#bottom#category#codepoint#codeextra#text% ... - \ifx\currentmathtext\empty + \ifx\m_math_stackers_text_middle\empty \setbox\scratchboxthree\emptyhbox \else

Thanks for looking into this, Wolfgang.

I made this change to math-stc.mkvi in the 2014.05.17 beta, then remade the MkIV format with

mtxrun --selfupdate mtxrun --generate luatools --generate context --make

context --make does a --generate so you can skip that and the luatools run is not needed either (selfupdate is seldom needed too)

Then I ran context on my test file (with 30000 as the \dorecurse count).

However, the offset underbraces remain. I'm attaching a image from p. 2128, by which point the offset is 20 or 30pt. I am fairly sure I am running the newly created version, because remaking the format included these lines:

open source > 1 > 256 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/math-stc.mkvi loading > ConTeXt Math Macros / Stackers close source > 1 > 256 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/math-stc.mkvi

and the log output from running the minimal.tex shows that the the fmt is indeed dated 2014.7.3:

mtx-context | run 1: luatex --fmt="/home/sanjoy/context/2014.05.17-beta/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en" --jobname="minimal" --lua="/home/sanjoy/context/2014.05.17-beta/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en.lui" --no-parse-first-line --c:currentrun=1 --c:fulljobname="./minimal.tex" --c:input="./minimal.tex" --c:kindofrun=1 --c:maxnofruns=8 "cont-yes.mkiv" This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/dev) (rev 4972) \write18 enabled. open source > 1 > 1 > /home/sanjoy/context/2014.05.17-beta/tex/texmf-context/tex/context/base/cont-yes.mkiv

ConTeXt ver: 2014.05.17 23:46 MKIV beta fmt: 2014.7.3 int: english/english

we get a .5pt height difference somewhere (nothing else different in a trace) ... smells like a luatex bug (testing that now) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Fri, Jul 4, 2014 at 12:37 AM, Sanjoy Mahajan wrote:

...

context --make does a --generate so you can skip that and the luatools run is not needed either (selfupdate is seldom needed too)

So 'context --make' is all one needs? I'll update the wiki.

...

we get a .5pt height difference somewhere (nothing else different in a trace) ... smells like a luatex bug (testing that now)

That is my suspicion as well. But I couldn't reproduce it with plain luatex. Though maybe I needed to use equation numbers, however that is done in plain tex (the context run doesn't show the problem without the \placeformula).

I suspect some internal state is getting slightly corrupted and getting progressively worse with the 30,000 repetitions.

I've been finding many unexplained 0.5pt or so differences, which may be related to the underbrace problem. The context, so to speak, is that I am putting in the last edits to the book, after having gone through and checked every page many times. So I want to ensure that none of the small changes has a big effect, but I don't want to check every page again (or I'll vomit).

Thus, I run my compare-pdfs.sh script on the previous and the latest pdf. It renders each page to a B/W png (using mupdf) and then compares corresponding pages using 'compare' (one of the ImageMagick tools). Any differing pixels are in red, and the shared pixels are in very light gray. Then all the pages with enough differences are viewed with 'feh', in order of most-to-least differences. And I check that nothing major has changed.

What I keep finding is that pages long after where I make a small change somehow get changed, with roughly one-pixel shifts to parts of the page. I'm attaching an example pixel diff. It shows changes on p.72 of the mss. However, the only change to the source code was on p.58 (and was designed not to change any page breaks, which it didn't). That same page (p.72) often gets similar changes with all kinds of small changes.

[Note to me: In case I need to recreate the source for debugging purposes, it's the change from rev 41ff54 to 2467d5]

In case the script is useful to others (earlier versions are in the contexttest repository somewhere), I am attaching it. I use it as follows with a bash command line. (The awk in the pipeline makes sure that only pages with some difference make it to the viewer.)

DPI=72 compare-pdfs.sh old.pdf new.pdf | tee book-print.compare72dpi | awk '{if (($3 > 0.02)) print;}' | sort -nr -k3 | awk '{print $1}' | xargs -r feh

Thank you for the script. Under linux you can also try diffpdf $> aptitude search diffpdf i diffpdf - compare two PDF files textually or visually

p diffpdf:i386 - compare two PDF files textually or visually -- luigi

On 7/4/2014 12:37 AM, Sanjoy Mahajan wrote:

...

context --make does a --generate so you can skip that and the luatools run is not needed either (selfupdate is seldom needed too)

So 'context --make' is all one needs? I'll update the wiki.

...

we get a .5pt height difference somewhere (nothing else different in a trace) ... smells like a luatex bug (testing that now)

That is my suspicion as well. But I couldn't reproduce it with plain luatex. Though maybe I needed to use equation numbers, however that is done in plain tex (the context run doesn't show the problem without the \placeformula).

I suspect some internal state is getting slightly corrupted and getting progressively worse with the 30,000 repetitions.

here it happens after a pagebreak (a few lines overflow are ok)

I've been finding many unexplained 0.5pt or so differences, which may be related to the underbrace problem. The context, so to speak, is that I am putting in the last edits to the book, after having gone through and checked every page many times. So I want to ensure that none of the small changes has a big effect, but I don't want to check every page again (or I'll vomit).

Thus, I run my compare-pdfs.sh script on the previous and the latest pdf. It renders each page to a B/W png (using mupdf) and then compares corresponding pages using 'compare' (one of the ImageMagick tools). Any differing pixels are in red, and the shared pixels are in very light gray. Then all the pages with enough differences are viewed with 'feh', in order of most-to-least differences. And I check that nothing major has changed.

What I keep finding is that pages long after where I make a small change somehow get changed, with roughly one-pixel shifts to parts of the page. I'm attaching an example pixel diff. It shows changes on p.72 of the mss. However, the only change to the source code was on p.58 (and was designed not to change any page breaks, which it didn't). That same page (p.72) often gets similar changes with all kinds of small changes.

[Note to me: In case I need to recreate the source for debugging purposes, it's the change from rev 41ff54 to 2467d5]

In case the script is useful to others (earlier versions are in the contexttest repository somewhere), I am attaching it. I use it as follows with a bash command line. (The awk in the pipeline makes sure that only pages with some difference make it to the viewer.)

DPI=72 compare-pdfs.sh old.pdf new.pdf | tee book-print.compare72dpi | awk '{if (($3 > 0.02)) print;}' | sort -nr -k3 | awk '{print $1}' | xargs -r feh

-Sanjoy

-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 7/4/2014 6:58 AM, Wolfgang Schuster wrote:

Am 04.07.2014 um 00:51 schrieb Hans Hagen mailto:pragma@wxs.nl>:

...

On 7/4/2014 12:37 AM, Sanjoy Mahajan wrote:

...

...

context --make does a --generate so you can skip that and the luatools run is not needed either (selfupdate is seldom needed too)

So 'context --make' is all one needs? I'll update the wiki.

...

we get a ..5pt height difference somewhere (nothing else different in a trace) .... smells like a luatex bug (testing that now)

That is my suspicion as well. But I couldn't reproduce it with plain luatex. Though maybe I needed to use equation numbers, however that is done in plain tex (the context run doesn't show the problem without the \placeformula).

I suspect some internal state is getting slightly corrupted and getting progressively worse with the 30,000 repetitions.

here it happens after a pagebreak (a few lines overflow are ok)

It does also depend on the width of the text because when I use 48pt in the example below the height of the brace is always the same but with 49pt the height increases after the first page.

\starttext

\dorecurse{100} {\startformula \mathord

{%\setbox\scratchbox\mathstylehbox{\Umathaccent\fam\zerocount"FE3DF{\hskip48pt}}%

\setbox\scratchbox\mathstylehbox{\Umathaccent\fam\zerocount"FE3DF{\hskip49pt}}% \the\userpageno:\the\ht\scratchbox}% \stopformula}

\stoptext

\starttext \dorecurse{1000} {\setbox\scratchbox\hbox{$\Umathaccent\fam\zerocount"FE3DF{\hskip49pt}$}% \the\ht\scratchbox\par} \stoptext that was the one i used when seeing some trigger from the page break (the two at the top of page 2 are the overshoot before tex triggers the page) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 7/4/2014 11:34 AM, Sanjoy Mahajan wrote:

Wolfgang Schuster writes:

...

It does also depend on the width of the text because when I use 48pt in the example below the height of the brace is always the same but with 49pt the height increases after the first page.

An interesting bug. That's also consistent with my experiments, in which

\underbrace{\hbox{yyyy yyyy}}_{\rho}

triggers the problem, but

\underbrace{\hbox{yyyy}}_{\rho}

does not.

Hi Sanjoy, After a day of testing, tracing, wondering etc Luigi and I figured out what was the problem it tooks us lots of debug prints -). Indeed there is a bug in luatex. An extensible is made up of a sequence char skip 0pt- xpt char skip 0pt- xpt char skip 0pt- xpt char And the code that determines the height/depth took the dimensions of skips into the equation while in fact glue has no such dimensions (leaders that are a special kind of glue have dimensions). Not the first place one looks for such an issue. Probably a side effect of copying a function and patching part. So, some random value (not entirely random as it depends on what mem area the glue node is allocated from and that is kind of stable in a run) was added which results in the effects you noticed. Because it is stable within a run (same successive errors) it doesn't result in oscillating (too many runs due to differences in results) and this is probably why it went unnoticed so long. Interesting. (Normally the pseudo random ht/dp were small.) So, once this fix moves upstream you can check your suspicious cases. Luigi will push it in our experimental branch and I suppose that Taco will then push it into trunk and after that you can compile or wait till the garden carries it. Thanks for noticing! Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Once it in the luatex trunk, I will check it right away.

I just compiled and installed luatex 0.79.1 r5033, remade the formats, and the result passes my underbrace tests (the 30,000 iterations and the examples in my book mss). Thank you for the great debugging. -Sanjoy

Wolfgang Schuster writes:

math-stc.mkvi:

% - \ifx\currentmathtext\empty + \ifx\m_math_stackers_text_middle\empty \setbox\scratchboxthree\emptyhbox

Is this change still also needed (in addition to the luatex fix)? -Sanjoy