Re: [Dev-luatex] nodelist based tex.saveboxresource

newer
luatex build breaks in current TL

older
Re: [Dev-luatex] nodelist based...

Hans Hagen

7 Aug 2018 7 Aug '18

11 a.m.

On 8/7/2018 7:23 AM, Marcel Krüger wrote:

...

---- On Fri, 03 Aug 2018 15:51:24 +0200 Hans Hagen wrote ----

...
On 8/3/2018 1:09 AM, Marcel Krüger wrote:

...
Hello,

I want to suggest extending the tex.saveboxresource interface to accept a hlist or vlist node as an alternative to a box number as first argument. Currently creating a boxresource from Lua node list requires saving the list in a TeX box register, only to delete that register again by tex.saveboxresource. A direct way to transform a h/vlist into a boxresource would make such code easier and avoid potential side effects from changing the TeX registers. An experimental patch with a suggested implementation is attached.

ok, makes sense (but not entirely ok that way but i can fix it)

...
I also attached a patch implementing a node.scan method which I think would be another nice addition for LuaTeX: It scans a box with rules similar to `\setbox0` and returns the node. This allows passing box-like material to LuaTeX without requiring box registers.

this is tricky stuff so i need to have a close look at side effects (it's also more relatex to token scanning so belongs in another namespace)

i'll have a go at it

Thank you very much. Just so you know while testing the new changes I noticed that the current trunk generates a warning

warning (pdf backend): dangling objects discarded

for *every* non-empty document.

I'm checking some backend issues and in the process that can happen. If trunk generates that then it's something else triggering it. Experimental is just that: experimental and things can change there stuff showing up there can even go. Also, if something is not documented, then don't use it. (This dangling is something i noticed and will be fixed, in addition to a possible accessor.)

...

Anyway there is another maybe more controversial feature I would like to see in LuaTeX: Could you add a callback replacing the built-in `do_delimiter`? This would allow creating big delimiter in more flexible ways than just stacking boxes, especially I think about drawing custom-size delimiters using mplib > I am aware that this could be archieved by using the `mlist_to_hlist` callback but I think experimenting with delimiter construction should not require rewriting the entire mathlist conversion in Lua.

Won't happen. For such fundamental extensions, changes, patches, adaptions one can use lua. In fact, drawing delimiters with e.g. mp can already be done for many years. It really makes no sense to replace each bit. We're now more than a decade in luatex and imo we can do whatever we want to extend in lua now. Also, given history it's nearly impossible to get an agreement on extensions that have hard coded behaviour. Probably the only bit that might evolve a bit is cleaning up the backend. (And I might start experimenting with a stripped down luatex follow up for context some day.)

...

Another option might be adding a `<function>` argument to node.mlist_to_hlist which, when not nil, is called to create delimiters. Won't happen either. Basically we're now at a stage where there will be no more extensions added. If needed there will be more parameters to replace hard coded values, but that's it.

Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Show replies by date

Marcel Krüger

16 Aug 16 Aug

4:59 p.m.

New subject: nodelist based tex.saveboxresource

---- On Tue, 07 Aug 2018 13:00:32 +0200 Hans Hagen wrote ----

...

On 8/7/2018 7:23 AM, Marcel Krüger wrote:

...
---- On Fri, 03 Aug 2018 15:51:24 +0200 Hans Hagen wrote ----

...
On 8/3/2018 1:09 AM, Marcel Krüger wrote:

...
Hello,

I want to suggest extending the tex.saveboxresource interface to accept a hlist or vlist node as an alternative to a box number as first argument. Currently creating a boxresource from Lua node list requires saving the list in a TeX box register, only to delete that register again by tex.saveboxresource. A direct way to transform a h/vlist into a boxresource would make such code easier and avoid potential side effects from changing the TeX registers. An experimental patch with a suggested implementation is attached.

ok, makes sense (but not entirely ok that way but i can fix it)

...
I also attached a patch implementing a node.scan method which I think would be another nice addition for LuaTeX: It scans a box with rules similar to `\setbox0` and returns the node. This allows passing box-like material to LuaTeX without requiring box registers.

this is tricky stuff so i need to have a close look at side effects (it's also more relatex to token scanning so belongs in another namespace)

i'll have a go at it

Thank you very much. Just so you know while testing the new changes I noticed that the current trunk generates a warning

warning (pdf backend): dangling objects discarded

for *every* non-empty document.

I'm checking some backend issues and in the process that can happen. If trunk generates that then it's something else triggering it.

Experimental is just that: experimental and things can change there stuff showing up there can even go. Also, if something is not documented, then don't use it. (This dangling is something i noticed and will be fixed, in addition to a possible accessor.)

I'm sorry, of course I meant experimental and not trunk.

...

...
Anyway there is another maybe more controversial feature I would like to see in LuaTeX: Could you add a callback replacing the built-in `do_delimiter`? This would allow creating big delimiter in more flexible ways than just stacking boxes, especially I think about drawing custom-size delimiters using mplib > I am aware that this could be archieved by using the `mlist_to_hlist` callback but I think experimenting with delimiter construction should not require rewriting the entire mathlist conversion in Lua.

Won't happen. For such fundamental extensions, changes, patches, adaptions one can use lua. In fact, drawing delimiters with e.g. mp can already be done for many years. It really makes no sense to replace each bit. We're now more than a decade in luatex and imo we can do whatever we want to extend in lua now. Also, given history it's nearly impossible to get an agreement on extensions that have hard coded behaviour.

I already expected a response along these lines. When you write that "drawing delimiters with e.g. mp can already be done" are you referring to a specific way? I experimented with this and could think of three approaches, but I'm not happy about any of them: - Rewriting `mlist_to_hlist` in Lua, at least the recursion into sub_mlists. I tried to avoid this because it looked like there are a lots of special cases and some internal state which would hard to track. - Use dummy delimiters marked with attributes and replace them after mlist to hlist conversion. This works quite well except that changing e.g. the width of vertical delimiters does not really work, especially when horizontal delimiters are involved too. - Do not use TeX delimiters (\left, \right, \middle, \radical, etc.) but emulate them using Lua/TeX macros. This wan't really an option because I had to be compatible with legacy input using the TeX functionality. Is there a better way I missed? For my current use-case I was able to solve my problems by using a patched LuaTeX version to accept glyph specific Lua functions in the `horiz_variants` and `vert_variants`, but I would love to see a nicer way which doesn't require tampering with the engine. On another note, I noticed that `\Uskewed` uses a delimiter for the middle character but it is always set with minimal height. This seems to be against the point of using a delimiter in the first place and it also isn't compatible with the handling of skewed fractions in other OpenType Math renderers like MS Word, so I wanted to ask if this is by design or if dynamic height of the fraction slash simply isn't implemented (yet)? Best regards Marcel Krüger

Hans Hagen

5:44 p.m.

New subject: nodelist based tex.saveboxresource

On 8/16/2018 6:59 PM, Marcel Krüger wrote:

...

...
...
Anyway there is another maybe more controversial feature I would like to see in LuaTeX: Could you add a callback replacing the built-in `do_delimiter`? This would allow creating big delimiter in more flexible ways than just stacking boxes, especially I think about drawing custom-size delimiters using mplib > I am aware that this could be archieved by using the `mlist_to_hlist` callback but I think experimenting with delimiter construction should not require rewriting the entire mathlist conversion in Lua.

Won't happen. For such fundamental extensions, changes, patches, adaptions one can use lua. In fact, drawing delimiters with e.g. mp can already be done for many years. It really makes no sense to replace each bit. We're now more than a decade in luatex and imo we can do whatever we want to extend in lua now. Also, given history it's nearly impossible to get an agreement on extensions that have hard coded behaviour.

I already expected a response along these lines. When you write that "drawing delimiters with e.g. mp can already be done" are you referring to a specific way?

the 'context' way which is not the 'latex' way so of not much relevance here as context does math a bit different

...

I experimented with this and could think of three approaches, but I'm not happy about any of them:

- Rewriting `mlist_to_hlist` in Lua, at least the recursion into sub_mlists. I tried to avoid this because it looked like there are a lots of special cases and some internal state which would hard to track.

you can indeed walk over that list and mess with specific nodes, you can do that before calling mlist_to_hlist or after (and then maybe fix soem things) .. in fact locating the delimiters in no big deal, and you could mark them beforehand (with an attribute or user node) and then afterwards mess a bit with the created boxes

...

- Use dummy delimiters marked with attributes and replace them after mlist to hlist conversion. This works quite well except that changing e.g. the width of vertical delimiters does not really work, especially when horizontal delimiters are involved too.

well, that's math ... heuristics ... interferences within fonts ... conflicting demands ... also depending on taste

...

- Do not use TeX delimiters (\left, \right, \middle, \radical, etc.) but emulate them using Lua/TeX macros. This wan't really an option because I had to be compatible with legacy input using the TeX functionality.

indeed, that's the most reasonable route .. often macro packages wrap stuff in macros anyway .. it also gives you precise control

...

Is there a better way I missed? For my current use-case I was able to solve my problems by using a patched LuaTeX version to accept glyph specific Lua functions in the `horiz_variants` and `vert_variants`, but I would love to see a nicer way which doesn't require tampering with the engine.

adding all kind of extra callbacks for handling such cases is opening cans of worms ... radicals, accents then follow too so we would have probably more that this hack ... in the end the tex math machinery is kind of a standard and we keep it that way ... after all it served the community well also, it is not the intention of the project to open up every tiny bit of tex and so for now mlist_to_hlist has to do (so far all i wanted to do could be done one wya or the other); this kind of opening up would mean tons of extra callbacks (which also adds overhead that we don't want) also, keep in mind that the more callbacks there are, the more users can kick in code and in the end all has to work together; so in fact, this kind of opening up and patching with lua code will backfire at some point ... interferences, expectations of how a list looks, etc etc ... i guess that it can be a macro package integrators nightmare

...

On another note, I noticed that `\Uskewed` uses a delimiter for the middle character but it is always set with minimal height. This seems to be against the point of using a delimiter in the first place and it also isn't compatible with the handling of skewed fractions in other OpenType Math renderers like MS Word, so I wanted to ask if this is by design or if dynamic height of the fraction slash simply isn't implemented (yet)?

by design ... these slanted (skewed) things are not really delimiters and most fonts don't even offer many variants (in fact, only a few symbols can really grow using the variants mechanism as slopes are not part of the game) also, middle is kind of special (introduced in etex btw) and conceptually fuzzy .. there is some more control in luatex but it will never be perfect anyway, concernign more callbacks, i'll keep it in mind when i look at math again, but any change in that department is not happening in 1.10 (one of these days we will relase 1.09 as prelude for a stable 1.10) and even if we would add some more control it would not be a quick hack either (as with many such mechanism probably a few years testing in context before settling on something would be part of the game) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hans Hagen

6:19 p.m.

New subject: nodelist based tex.saveboxresource

Hi, btw, you probably noticed that mlist_to_hlist is called recursively so you might use that fact conceptually one could have callbacks at the make_* level (the wrapping things) which is kind of clean but that has not much use as work happens in those and duplicating all that happens in there in lua makes no sense a do_delimiter callback is only a partial solution as it is used in several places and also works in tandem with other constructs (rules for instance); also, you'd always have to keep in mind that other mechanisms expect (for instance) attributes to be retained, character properties to migrate to unicode (cut and paste), etc, but that can be dealt with, in that sense the name 'delimiter' is somewhat confusing (fwfw, in context we hardly use these constructs because we want to have more control and do fancy things) anyway, a dodelimiter callback is in some sense not so much a delimiter one, but more a constructor one as it kind of relates to fonts and not so much to math ... basically it could be something "create_extensible" or so which is then triggered inside e.g. do_delimiter i'll play with that a bit when i have time (i.e. what is the context we need to pass and what do we want back and what wrapup afterwards is needed) to see if it makes sense at all Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Marcel Krüger

7:08 p.m.

New subject: nodelist based tex.saveboxresource

---- On Thu, 16 Aug 2018 20:19:16 +0200 Hans Hagen wrote ----

...

Hi,

btw, you probably noticed that mlist_to_hlist is called recursively so you might use that fact

I had some experiments in this area but it is quite hard to avoid side effects here. Anyway it seems to be the most promising way. Related to this: Strict reading of the manual would imply that node.mlist_to_hlist only accepts 'display' or 'text' as style parameter. While you could emulate everything else by adding style nodes, the implementation accepts other styles too and it seemed to make sense to pass the specific style directly. Is this officially supported?

...

conceptually one could have callbacks at the make_* level (the wrapping things) which is kind of clean but that has not much use as work happens in those and duplicating all that happens in there in lua makes no sense

a do_delimiter callback is only a partial solution as it is used in several places and also works in tandem with other constructs (rules for instance); also, you'd always have to keep in mind that other mechanisms expect (for instance) attributes to be retained, character properties to migrate to unicode (cut and paste), etc, but that can be dealt with, in that sense the name 'delimiter' is somewhat confusing

(fwfw, in context we hardly use these constructs because we want to have more control and do fancy things)

I certainly have to take me some time to look more into ConTeXt but I generally try to avoid using too much GPL licensed stuff... Anyway, that's completely off-topic.

...

anyway, a dodelimiter callback is in some sense not so much a delimiter one, but more a constructor one as it kind of relates to fonts and not so much to math ... basically it could be something "create_extensible" or so which is then triggered inside e.g. do_delimiter

i'll play with that a bit when i have time (i.e. what is the context we need to pass and what do we want back and what wrapup afterwards is needed) to see if it makes sense at all

Just in case you are interested in my code I attached a patch I am currently experimenting with which moves this completely into the font and triggers it in `get_delim_box` to avoid interference for "easy" delimiters. It a hack which probably fails completely if you try dumping the font during format generation and it does some mostly untested and improvised reference counting, but for easy cases it seems to work quite reliable. Anyway I agree that adding that adding anything here might not be worth it so I will see if I continue with using custom patches or if I find a sufficiently nice `mlist_to_hlist` solution.

...

Hans

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Best regards Marcel

Hans Hagen

17 Aug 17 Aug

8:03 a.m.

New subject: nodelist based tex.saveboxresource

On 8/16/2018 9:08 PM, Marcel Krüger wrote:

...

---- On Thu, 16 Aug 2018 20:19:16 +0200 Hans Hagen wrote ----

...
Hi,

btw, you probably noticed that mlist_to_hlist is called recursively so you might use that fact

I had some experiments in this area but it is quite hard to avoid side effects here. Anyway it seems to be the most promising way. Related to this: Strict reading of the manual would imply that node.mlist_to_hlist only accepts 'display' or 'text' as style parameter. While you could emulate everything else by adding style nodes, the implementation accepts other styles too and it seemed to make sense to pass the specific style directly. Is this officially supported?

sure as it is used recursively for all styles

...

Just in case you are interested in my code I attached a patch I am currently experimenting with which moves this completely into the font and triggers it in `get_delim_box` to avoid interference for "easy"

last night i came to the conclusion that the only feasible plug is in the construction of a compound glyph but then we also need to provide it as helper function i.e. it is more a font related thing than math related (even if it gets triggered by a delimiter

...

delimiters. It a hack which probably fails completely if you try dumping the font during format generation and it does some mostly untested and improvised reference counting, but for easy cases it seems to work quite reliable.

best forget about dumping that kind of font stuff .. lua states are not dumped and anything lua related to fonts neither ... Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Marcel Krüger

10:11 a.m.

New subject: nodelist based tex.saveboxresource

---- On Fri, 17 Aug 2018 10:03:13 +0200 Hans Hagen wrote ----

...

last night i came to the conclusion that the only feasible plug is in the construction of a compound glyph but then we also need to provide it as helper function i.e. it is more a font related thing than math related (even if it gets triggered by a delimiter

This sounds similar to my last approach. If you can select using Lua or the builtin functionality on a per-glyph or even per-font basis, the helper should not be that important, even if providing it might be useful anyway.

...

...
delimiters. It a hack which probably fails completely if you try dumping the font during format generation and it does some mostly untested and improvised reference counting, but for easy cases it seems to work quite reliable.

best forget about dumping that kind of font stuff .. lua states are not dumped and anything lua related to fonts neither ...

This was more about printing an error if someone tries it instead of dumping a font which calls random code whenever a delimiter is used... Marcel

2319

Age (days ago)

2329

Last active (days ago)

List overview

Download

6 comments

2 participants

participants (2)

Hans Hagen
Marcel Krüger