Re: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

newer
adding time to signatures

older
Re: LuaMetaTeX accepts `;` as part...

Hans Hagen

8 Nov 2024 8 Nov '24

3:27 p.m.

On 11/8/2024 9:53 AM, Joseph Wright wrote:

...

Hello both,

On 06/11/2024 15:28, Hans Hagen wrote:

...
On 11/6/2024 12:37 PM, Vítek Starý Novotný wrote:

...
...
Anyway, today I'm not in the mood to ponder semi colons, as it's the least of my worries,

I am very sorry to bother you and other ConTeXt devs with this. If it makes it feel any better, the LaTeX folks seem equally annoyed with me.

...
I'm not sure what "just %" means here.

These were Joseph Wright's words. I assume Joseph was referring to the fact that in e-TeX and LuaTeX, including a comment after a `\numexpr` causes the comment to be ignored by TeX, similarly to how in LuaLaTeX, including a semicolon causes parts of the following text to be ignored.

It's not a comment - it's that the standard e-TeX implementation for \numexpr only accept a very limited number of characters, and any characters that are non-expandable and not in that set are treated as end-of-expression.

sure, that's what i thought but it's good to have that thread fixed

...

...
Well, that's just numexpr then, but how about:

\def\fool{\hskip10pt plus 0pt}

oeps \fool me

oeps \fool minus me

This whole lookahead scanning is fragile by design and just like one should define

\def\fool{\hskip10pt plus 0pt\relax}

Most of the time I agree, but \relax is a lot longer than (say) ; and if you are writing a complex parser that can be handy.

[In the end, if ; stays as part of the extended \numexpr syntax for LuaMetaTeX, at some stage I will have to re-work the code causing the issue here to use another token in place of ; - \relax (in our case \scan_stop:) really is a bit long, so probably something like @ is the viable change.]

sure, but this expl stuff si pretty verbose and long winding anyway (or that's what i remember from some BT presentation)

...

...
one should also make sure an expression doesn't scan ahead, if only because it will expand whats coming next; basically \relax or matching () should have been mandate and it's a design flaw.

Arguably yes but e-TeX was finalised in 1999 - that was an discussion for a previous millennium (at least from a plain/LaTeX point of view) :)

as much of tex ..

...

...
As mentioned there, we also cleaned up some function names (or moved some to another namespace); there's a reason why luatex was frozen and luametatex was the follow up.

Of course - but one day one might imagine LuaMetaTeX will more something that makes sense for non-ConTeXt users too. (Plain and OPTeX as well as LaTeX, I imagine.)

i can add \etexexprmode=1 that ignores : and ; operations (not much overhead) as long as it doesn't trigger all kind of "can you also do this or that" demands to make < 1999 inspired code happy) if course \etexexprmode defaults to zero Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Show replies by date

Vítek Starý Novotný

11 Nov 11 Nov

12:17 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

On Fri, Nov 08, 2024 at 03:27:16PM +0100, Hans Hagen wrote:

...

On 11/8/2024 9:53 AM, Joseph Wright wrote:

...
It's not a comment - it's that the standard e-TeX implementation for \numexpr only accept a very limited number of characters, and any characters that are non-expandable and not in that set are treated as end-of-expression.

sure, that's what i thought but it's good to have that thread fixed

Thank you, Joseph, for clarifying, and apologies for any confusion from my end. On Wed, Nov 06, 2024 at 04:28:04PM +0100, Hans Hagen wrote:

...

On 11/6/2024 12:37 PM, Vítek Starý Novotný wrote:

...
The expl3 programming language from LaTeX3 can be used in other formats as well, including plain TeX and ConTeXt with LuaTeX, among others. Packages (such as my Markdown package for TeX) use expl3 as a format-independent programming layer.

So maybe it's (at least for now) good to stick to luatex/mkiv, also because if markdown is the input there is not that much to gain from luametatex

Using MkIV is feasible for me personally, but we do have ConTeXt users interested in using LMTX with Markdown for text formatting. I’d like to support that if possible, particularly as MkIV may eventually phase out. That said, it's becoming clear that using expl3 with non-LaTeX formats presents challenges. Performance is one issue; expl3 isn't part of the format files for other formats and needs initialization at the start of each run, which adds overhead. More significantly, it often doesn't align with the philosophy of certain formats. For instance, OPTeX prioritizes minimalism and closeness to plain TeX, with more difficult tasks being offloaded to Lua code, making expl3 feel somewhat out of place. ConTeXt, likewise, is Lua-focused, with users more accustomed to Lua than to TeX macros. For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt. Best regards, Vitek

Henning Hraban Ramm

12:56 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

Am 11.11.24 um 12:17 schrieb Vítek Starý Novotný:

...

For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt.

I though ConTeXt had a markdown module, but that refers to your https://ctan.org/pkg/markdown, oops. Otherwise you can use Pandoc’s ConTeXt backend; AFAIK that’s what most ConTeXt & Markdown users do. Hraban

Hans Hagen

1:27 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

On 11/11/2024 12:56 PM, Henning Hraban Ramm wrote:

...

Am 11.11.24 um 12:17 schrieb Vítek Starý Novotný:

...
For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt.

I though ConTeXt had a markdown module, but that refers to your https:// ctan.org/pkg/markdown, oops.

at some point there was a third party markdown lua module and that was dead slow so i redid that in lpeg and fed it back; it probably is arround somehwere but i have no clue how markdown evolved (can't keep track of everything), if there was need i could look into it Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Vítek Starý Novotný

2 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

The history is a bit convoluted. Let me see if I can help untangle it: - In 2006, Niklas Frykholm created `markdown.lua` [1], a regex-based Markdown parser written in Lua. - In 2009, John MacFarlane, the founder of Pandoc, created Lunamark [2], an LPEG-based Markdown parser that was ca three orders of magnitude faster than `markdown.lua`. Around 2011, there is a bunch of commits with speed optimizations by Hans. - In 2016, I created the Markdown Package for TeX [3] as a fork of Lunamark [3]. Initially, I would contribute fixes upstream and take them downstream. In 2023, there was a significant rewrite to support the CommonMark standard [4], which makes our code base largely incompatible with Lunamark. We are actively developed with monthly releases. Confusingly, the Markdown Package for TeX contains a file titled `markdown.lua` even though there is no relation to Niklas Frykholm's code. [1]: http://www.frykholm.se/files/markdown.lua [2]: https://github.com/jgm/lunamark [3]: https://github.com/witiko/markdown [4]: https://spec.commonmark.org/ I hope this clears things up. Best, Vitek On Mon, Nov 11, 2024 at 01:27:18PM +0100, Hans Hagen wrote:

...

On 11/11/2024 12:56 PM, Henning Hraban Ramm wrote:

...
Am 11.11.24 um 12:17 schrieb Vítek Starý Novotný:

...
For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt.

I though ConTeXt had a markdown module, but that refers to your https:// ctan.org/pkg/markdown, oops.

at some point there was a third party markdown lua module and that was dead slow so i redid that in lpeg and fed it back; it probably is arround somehwere but i have no clue how markdown evolved (can't keep track of everything), if there was need i could look into it

Hans

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl ----------------------------------------------------------------- _______________________________________________ dev-context mailing list -- dev-context@ntg.nl To unsubscribe send an email to dev-context-leave@ntg.nl

luigi scarso

5:37 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

On Mon, 11 Nov 2024 at 14:00, Vítek Starý Novotný wrote:

...

The history is a bit convoluted. Let me see if I can help untangle it:

- In 2006, Niklas Frykholm created `markdown.lua` [1], a regex-based Markdown parser written in Lua.

- In 2009, John MacFarlane, the founder of Pandoc, created Lunamark [2], an LPEG-based Markdown parser that was ca three orders of magnitude faster than `markdown.lua`. Around 2011, there is a bunch of commits with speed optimizations by Hans.

- In 2016, I created the Markdown Package for TeX [3] as a fork of Lunamark [3]. Initially, I would contribute fixes upstream and take them downstream. In 2023, there was a significant rewrite to support the CommonMark standard [4], which makes our code base largely incompatible with Lunamark. We are actively developed with monthly releases.

Confusingly, the Markdown Package for TeX contains a file titled `markdown.lua` even though there is no relation to Niklas Frykholm's code.

[1]: http://www.frykholm.se/files/markdown.lua [2]: https://github.com/jgm/lunamark [3]: https://github.com/witiko/markdown [4]: https://spec.commonmark.org/

I hope this clears things up.

https://github.com/commonmark/cmark seems to be quite fast -- there is an output format for latex, an output format for context mkiv might make sense (just for (my) records) -- luigi

Henning Hraban Ramm

5:55 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

Am 11.11.24 um 17:37 schrieb luigi scarso:

...

On Mon, 11 Nov 2024 at 14:00, Vítek Starý Novotný mailto:witiko@mail.muni.cz> wrote:

The history is a bit convoluted. Let me see if I can help untangle it:

- In 2006, Niklas Frykholm created `markdown.lua` [1], a regex-based Markdown parser written in Lua.

- In 2009, John MacFarlane, the founder of Pandoc, created Lunamark [2], an LPEG-based Markdown parser that was ca three orders of magnitude faster than `markdown.lua`. Around 2011, there is a bunch of commits with speed optimizations by Hans.

- In 2016, I created the Markdown Package for TeX [3] as a fork of Lunamark [3]. Initially, I would contribute fixes upstream and take them downstream. In 2023, there was a significant rewrite to support the CommonMark standard [4], which makes our code base largely incompatible with Lunamark. We are actively developed with monthly releases.

Confusingly, the Markdown Package for TeX contains a file titled `markdown.lua` even though there is no relation to Niklas Frykholm's code.

[1]: http://www.frykholm.se/files/markdown.lua [2]: https://github.com/jgm/lunamark https://github.com/jgm/lunamark [3]: https://github.com/witiko/markdown <https://github.com/ witiko/markdown> [4]: https://spec.commonmark.org/ https://spec.commonmark.org/

I hope this clears things up.

https://github.com/commonmark/cmark https://github.com/commonmark/cmark seems to be quite fast -- there is an output format for latex, an output format for context mkiv might make sense

See also Massimiliano’s articles about his internal-Pandoc-format editor in last year’s CGJ: https://yemaya.fiee.net/s/DjJBtnTFedC9zL8 Hraban

Hans Hagen

1:04 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

On 11/11/2024 12:17 PM, Vítek Starý Novotný wrote:

...

On Fri, Nov 08, 2024 at 03:27:16PM +0100, Hans Hagen wrote:

...
On 11/8/2024 9:53 AM, Joseph Wright wrote:

...
It's not a comment - it's that the standard e-TeX implementation for \numexpr only accept a very limited number of characters, and any characters that are non-expandable and not in that set are treated as end-of-expression.

sure, that's what i thought but it's good to have that thread fixed

Thank you, Joseph, for clarifying, and apologies for any confusion from my end.

On Wed, Nov 06, 2024 at 04:28:04PM +0100, Hans Hagen wrote:

...
On 11/6/2024 12:37 PM, Vítek Starý Novotný wrote:

...
The expl3 programming language from LaTeX3 can be used in other formats as well, including plain TeX and ConTeXt with LuaTeX, among others. Packages (such as my Markdown package for TeX) use expl3 as a format-independent programming layer.

So maybe it's (at least for now) good to stick to luatex/mkiv, also because if markdown is the input there is not that much to gain from luametatex

Using MkIV is feasible for me personally, but we do have ConTeXt users interested in using LMTX with Markdown for text formatting. I’d like to support that if possible, particularly as MkIV may eventually phase out.

We use mkiv as benchmark for testing luatex so it's there to stay. We don't add new functionality but do fix issues. There are still some files shared between mkiv and mkxl (like font definitions and lua utility files). In fact, we also keep mkii around because we need to check out pdftex (and xetex) occasionally. Of course, when do some reasons the engines change issues might show up and dropping might be the only sollution.

...

That said, it's becoming clear that using expl3 with non-LaTeX formats presents challenges. Performance is one issue; expl3 isn't part of the format files for other formats and needs initialization at the start of each run, which adds overhead. More significantly, it often doesn't align with the philosophy of certain formats. For instance, OPTeX prioritizes minimalism and closeness to plain TeX, with more difficult tasks being offloaded to Lua code, making expl3 feel somewhat out of place. ConTeXt, likewise, is Lua-focused, with users more accustomed to Lua than to TeX macros.

That's not true. Most users use the tex input language. They might kick in soem metapost (metafun) and lua, or use a mix with lua when data has to be massaged, but coding mostly text document input in a programming language like lua is counter productive and especially for more complex and rich coding also a pain (think passing arguments and evaluation in a function based language with multiple layers of nesting).

...

For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt.

I don't know. It can be kind of amusing to see comments wrt context and luatex (i'm not talking of you here). For instance I remember a comparison in TB between optex being minimal (comparing format file size) while optex loads the fontloader and other stuff at runtime (which size is then ignored) and of course the macro base (i assume) will grow over time too and that all then makes the original comparisons and arguments kind of silly. The same is actually true for arguments for using latex over context ... successive past arguments get invalidated but naratives stay arround. A lot of weird and inconstent politics can enter tex related discussions and those could never really interest me, especially not when (technical) arguments make no sense and can easily be countered (but why waste time on it; it's not my duty to debunk). Anyway, in order to test this ; issue I downloaded the expl generic file and (with a change in the lua file -- different function) it could be loaded but esp the unicode part takes quite some time, likely more than users will accept in an edit-run cycle that normally takes a few seconds per run. I suppose that over time more will get loaded and that for sure will start clashing with macro package code / approaches so that might something that you need to keep an eye on: reduncancy, clashes, bloat, runtime, and whatever bothers the mkdown module. I can't comment on what gets and has to be loaded (apart fromn runtime) but i can imagine that it takes some effort from your end to make all macro packages happy. Of course, when you need some specific helpers we can provide them, Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Vítek Starý Novotný

1:41 p.m.

New subject: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

On Mon, Nov 11, 2024 at 01:04:08PM +0100, Hans Hagen wrote:

...

On 11/11/2024 12:17 PM, Vítek Starý Novotný wrote:

...
ConTeXt, likewise, is Lua-focused, with users more accustomed to Lua than to TeX macros.

That's not true. Most users use the tex input language. They might kick in soem metapost (metafun) and lua, or use a mix with lua when data has to be massaged, but coding mostly text document input in a programming language like lua is counter productive and especially for more complex and rich coding also a pain (think passing arguments and evaluation in a function based language with multiple layers of nesting).

I am sure they use TeX for the document markup but generally, ConTeXt users seem more comfortable setting up code that calls out to Lua and to external tools. In contrast, LaTeX users seem to rely more on packages and expect TeX code with little-to-no visible Lua code. Therefore, whereas a LaTeX user might expect the `markdown` package to provide an environment \begin{markdown} ... \end{markdown} that does the expected thing (even with legacy engines such as pdfTeX), a ConTeXt user can be more reasonably expected to set up a buffer that will filter the buffered text through the library `markdown.lua` themselves. It helps that ConTeXt provides format-level primitives such as buffers, whereas LaTeX seems more focused on the division of labor between programmers and authors, where authors should mostly rely on packages and only ever use high-level document markup. Best, Vitek

...

Hans

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Age (days ago)

Last active (days ago)

List overview

Download

8 comments

4 participants

participants (4)

Hans Hagen
Henning Hraban Ramm
luigi scarso
Vítek Starý Novotný

Re: LuaMetaTeX accepts `;` as part of `\numexpr`, which breaks LaTeX3

tags

participants (4)