On 7/7/2022 3:15 AM, Max Chernoff via ntg-context wrote:

Hi,

With the latest upload, "tex.linebreak" doesn't return an "info" table.

When running this code:

    \startluacode         function test(head)             local new_head, info = tex.linebreak(node.copylist(head))

            print(head, type(head), info, type(info))

            if info == nil then                 print "!!! info is nil !!!"             end

            return head         end

        nodes.tasks.appendaction("processors", "after", "test")     \stopluacode

    \starttext         Hello!     \stoptext

with the latest upload I get (also note the small typo in "rigthfill"):

    luatex warning  > linebreak: no [ leftinit | rightinit | leftfill | rigthfill] expected        2353 : par vmodepar> userdata        nil     nil     !!! info is nil !!!        2299 : glue userskip> userdata        nil     nil     !!! info is nil !!!

but with live.contextgarden.net I get:

    luatex warning  > tex: left parfill skip is gone     Underfull \hbox (badness 10000) in paragraph at lines 20--1         929 : par vmode_par>    userdata table: 0x39a050f7000    table        1561 : glue userskip>    userdata nil    nil     !!! info is nil !!!

This is because you need to pass a list that conforms to what the builder expects and the callback that you use doesn't do that for you (after all, it also gets hbox content). There is no 'callback sequence handler' for the par builder (currently i see no need for it, also given the extra overhead involved) but this what what you can do: \startluacode function builders.paragraphs.constructors.methods.preroll_a(head) local result, info = tex.linebreak(head) tex.prevdepth = info.prevdepth tex.prevgraf = info.prevgraf return result end function builders.paragraphs.constructors.methods.preroll_b(head) local result, info = tex.linebreak(nodes.nuts.copylist(head)) inspect(info) return true end \stopluacode \defineparbuilder[preroll_a] \defineparbuilder[preroll_b] \starttext \setmainparbuilder[default] \input tufte \par \input tufte \page \setmainparbuilder[preroll_a] \input tufte \par \input tufte \page \setmainparbuilder[preroll_b] \input tufte \par \input tufte \page \stoptext The tex.linebreak function gets the properly prepared paragraph and can handle it then. On the to do is a to add a 'prepare' helper that adds the mandate nodes (par fillers etc) but even then one has to be careful where linebreak hackery is applied. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On 7/8/2022 9:40 AM, Max Chernoff via ntg-context wrote:

...

This is because you need to pass a list that conforms to what the builder expects and the callback that you use doesn't do that for you (after all, it also gets hbox content).

Isn't "processors/after" the same as "pre_linebreak_filter"? I thought that only "hpack_filter" gets \hbox content? Anyways, my actual function has this guard at the very start:

    if (head.id ~= par_id and context) or -- Ensure that we were actually given a par         status.output_active or -- Don't run during the output routine         tex.nest.ptr > 1 -- Don't run inside boxes     then         return head     end

so I think that I'm only processing actual top-level paragraphs here.

...

There is no 'callback sequence handler' for the par builder (currently i see no need for it, also given the extra overhead involved) but this what what you can do:

\startluacode      function builders.paragraphs.constructors.methods.preroll_a(head)          local result, info = tex.linebreak(head)          tex.prevdepth = info.prevdepth          tex.prevgraf = info.prevgraf          return result      end      function builders.paragraphs.constructors.methods.preroll_b(head)          local result, info = tex.linebreak(nodes.nuts.copylist(head))          inspect(info)          return true      end \stopluacode

\defineparbuilder[preroll_a] \defineparbuilder[preroll_b]

\starttext      \setmainparbuilder[default]   \input tufte \par \input tufte \page      \setmainparbuilder[preroll_a] \input tufte \par \input tufte \page      \setmainparbuilder[preroll_b] \input tufte \par \input tufte \page \stoptext

I think that that code is for replacing the linebreaker entirely, like with using "linebreak_filter"? My goal isn't to replace the linebreaker; I just want to be able to inspect the paragraph before it is broken, without modifying anything.

as long as you return true the built in will do the work but you can still do the preroll so basically nothing gets replaced .. it's just a more natural hook;l keep in mind that if someone replaces the linebreak your code woudl fail to do its work anyway

What I'm trying to do *very* roughly looks like the following:

    paragraphs = {}     attribute = 1234

    function pre_linebreak_filter(head)         if head.id ~= node.id"par" then             return head         end

        local nat_node, nat_info = tex.linebreak(node.copylist(head))         node.freelist(nat_node)

        local long_node, long_info = tex.linebreak(             node.copylist(head), {looseness = 1}         )

        if long_info.prevgraf == nat_info.prevgraf + 1 then             table.insert(paragraphs, long_node)         end

        return head     end

    function post_linebreak_filter(head)         node.setattribute(head, attribute, #paragraphs)         node.setattribute(node.slide(head), attribute, #paragraphs)

        return head     end

    function pre_output_filter(head)         if tex.outputpenalty ~= tex.widowpenalty then             return head         end

        -- Pick a paragraph from `paragraphs` somehow

        -- Replace that paragraph on the page with the one         -- from `paragraphs`

        -- Move the last line of the page onto the top of         -- tex.lists.contributehead

        return head     end

(The full implementation is in the module "lua-widow-control" on CTAN, TeX Live, and modules.contextgarden.net, or directly at "https://github.com/gucci-on-fleek/lua-widow-control/blob/master/source/lua-w...". It's pretty long though, so I'm just trying to summarize here.)

This works pretty well with Plain LuaTeX, LuaLaTeX, OpTeX, MkIV, and MkXL before the latest upload, but something broke with the latest upload in MkXL. I understand that I'm mucking around with volatile interfaces, and I have no problem making a bunch of changes whenever the engine/format changes; the problem is that I'm not too sure what changed in the engine, so I don't know what I need to change in my code.

the tricky part is probably to make sure that it doesn't interfere with what other functions hooked into the callbacks are doing because who knows what gets injected in those lists

...

On the to do is a to add a 'prepare' helper that adds the mandate nodes (par fillers etc)

Maybe that's all that I need? If that's the case, I have no problem coding my own "prepare helper" if you think that it'll be awhile before you get around to it; the problem is that I'm not entirely sure what nodes I would need to add. These new nodes aren't added until after "pre_linebreak_filter", and they're gone after the linebreaker runs, so I can't inspect a "regular" paragraph to see where these nodes belong.

there will be (untested on what you do) tex.preparelinebreak(par) that will add the additional stuff needed (and bark if something is there already). btw, a par builder - even in luatex - needs a parfillskip to properly do its work, some penalty and in the case of context lmtx also a leftparfill skip and optionally initleft and right skip Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On 7/7/2022 10:48 PM, Aditya Mahajan via ntg-context wrote:

On Wed, 6 Jul 2022, Hans Hagen via ntg-context wrote:

...

We're pleased to announce the first official release of the new math code. We're talking of:

-- a reworked math engine (which happened stepwise so some already trickled into previous uploads)

-- settling on one model for display math but with alignment variants as well as improving inline math (e.g. wrt linebreaks)

-- redoing bit of the (sub) numbering

-- improving the alignment features (the traditional one and the text/page mode one)

-- fixing the math font using all kind of tweaks in the goodie file

-- providing means to annotate formulas

-- we ditched support for \over simply because it is too fragile wrt spacing (one can \usemodule[oldmath] to get it back

-- we might move some more alien stuff to m-oldmath in due time

I played around with it a bit, and everything looks really nice!

Thanks. Btw, you can run with \showmakeup[mathglue] % \mathspacingmode=1 to see what math atoms are 'seen' (the spacingmode also injects zero spaces) (there are other tracers too but this is the one we use most) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

I just found that \choose has been replaced with \binom (binomial coefficient): \starttext \meaning\choose \meaning\binom $\binom{n}{k}$ \stoptext Sorry for complaining… :-)

On 9 Jul 2022, at 00:34, Otared Kavian wrote:

...

On 6 Jul 2022, at 22:10, Hans Hagen via ntg-context wrote:

Hi,

We're pleased to announce the first official release of the new math code. We're talking of:

[…]

Dear Hans and Mikael،

Thank you very much for the new upload and the new stuff for tyepsetting maths. I have been playing around with the features presented in the example file sent by Mikael, and I must say, among other things you have implemented, the constructs \bearkhere and \skiphere, and friends, are absolutely great, brilliant and so simple! I wonder why such a simple and elegant solution was not around before…

As a plain TeX traditionalist I am a little bit sad to let the \over construct be ditched, but my sadness did not last too long.

It seems that the \choose construct has been ditched but I did not guess what has replaced it (I guess the reason is the same for ditching \over, since the construct used to be ${n \choose k}$ and this is not what one wants to have in a macro).

Best regards: Otared

On 7/9/2022 12:34 AM, Otared Kavian via ntg-context wrote:

...

On 6 Jul 2022, at 22:10, Hans Hagen via ntg-context wrote:

Hi,

We're pleased to announce the first official release of the new math code. We're talking of:

[…]

Dear Hans and Mikael،

Thank you very much for the new upload and the new stuff for tyepsetting maths. I have been playing around with the features presented in the example file sent by Mikael, and I must say, among other things you have implemented, the constructs \bearkhere and \skiphere, and friends, are absolutely great, brilliant and so simple! I wonder why such a simple and elegant solution was not around before…

they were to some extend (in split=text mode) but never advocated but we added some to it as well as made it work with diferent alignments (most noticeably slanted) and number placement (overflow)

As a plain TeX traditionalist I am a little bit sad to let the \over construct be ditched, but my sadness did not last too long.

just think of the side effects: {a}\over{y} + z y and z are combined so one need to add { } {{a}\over{y}} thenm infleuncing styles becomes trickier too; the lot becomes an ord (ok, not in lmtx, where it becomes a fraction atom) and when you compare {{a}\over{y}} \frac{a}{b} you even save two keystrokes -) it's also not easy to control spacing etc because one cannot redefine over; actually fractions uses \U[..over|above|..] which takes two arguments (forward scanning)

It seems that the \choose construct has been ditched but I did not guess what has replaced it (I guess the reason is the same for ditching \over, since the construct used to be ${n \choose k}$ and this is not what one wants to have in a macro).

some of these were just old school simple things but we have better alternatives; and if not: just specify what you need and we will come up with something (configurable) (of course one can just define the old schoose) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------