On 11/11/2024 12:56 PM, Henning Hraban Ramm wrote:

Am 11.11.24 um 12:17 schrieb Vítek Starý Novotný:

...

For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt.

I though ConTeXt had a markdown module, but that refers to your https:// ctan.org/pkg/markdown, oops.

at some point there was a third party markdown lua module and that was dead slow so i redid that in lpeg and fed it back; it probably is arround somehwere but i have no clue how markdown evolved (can't keep track of everything), if there was need i could look into it Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On Mon, 11 Nov 2024 at 14:00, Vítek Starý Novotný wrote:

The history is a bit convoluted. Let me see if I can help untangle it:

- In 2006, Niklas Frykholm created `markdown.lua` [1], a regex-based Markdown parser written in Lua.

- In 2009, John MacFarlane, the founder of Pandoc, created Lunamark [2], an LPEG-based Markdown parser that was ca three orders of magnitude faster than `markdown.lua`. Around 2011, there is a bunch of commits with speed optimizations by Hans.

- In 2016, I created the Markdown Package for TeX [3] as a fork of Lunamark [3]. Initially, I would contribute fixes upstream and take them downstream. In 2023, there was a significant rewrite to support the CommonMark standard [4], which makes our code base largely incompatible with Lunamark. We are actively developed with monthly releases.

Confusingly, the Markdown Package for TeX contains a file titled `markdown.lua` even though there is no relation to Niklas Frykholm's code.

[1]: http://www.frykholm.se/files/markdown.lua [2]: https://github.com/jgm/lunamark [3]: https://github.com/witiko/markdown [4]: https://spec.commonmark.org/

I hope this clears things up.

https://github.com/commonmark/cmark seems to be quite fast -- there is an output format for latex, an output format for context mkiv might make sense (just for (my) records) -- luigi

On 11/11/2024 12:17 PM, Vítek Starý Novotný wrote:

On Fri, Nov 08, 2024 at 03:27:16PM +0100, Hans Hagen wrote:

...

On 11/8/2024 9:53 AM, Joseph Wright wrote:

...

It's not a comment - it's that the standard e-TeX implementation for \numexpr only accept a very limited number of characters, and any characters that are non-expandable and not in that set are treated as end-of-expression.

sure, that's what i thought but it's good to have that thread fixed

Thank you, Joseph, for clarifying, and apologies for any confusion from my end.

On Wed, Nov 06, 2024 at 04:28:04PM +0100, Hans Hagen wrote:

...

On 11/6/2024 12:37 PM, Vítek Starý Novotný wrote:

...

The expl3 programming language from LaTeX3 can be used in other formats as well, including plain TeX and ConTeXt with LuaTeX, among others. Packages (such as my Markdown package for TeX) use expl3 as a format-independent programming layer.

So maybe it's (at least for now) good to stick to luatex/mkiv, also because if markdown is the input there is not that much to gain from luametatex

Using MkIV is feasible for me personally, but we do have ConTeXt users interested in using LMTX with Markdown for text formatting. I’d like to support that if possible, particularly as MkIV may eventually phase out.

We use mkiv as benchmark for testing luatex so it's there to stay. We don't add new functionality but do fix issues. There are still some files shared between mkiv and mkxl (like font definitions and lua utility files). In fact, we also keep mkii around because we need to check out pdftex (and xetex) occasionally. Of course, when do some reasons the engines change issues might show up and dropping might be the only sollution.

That said, it's becoming clear that using expl3 with non-LaTeX formats presents challenges. Performance is one issue; expl3 isn't part of the format files for other formats and needs initialization at the start of each run, which adds overhead. More significantly, it often doesn't align with the philosophy of certain formats. For instance, OPTeX prioritizes minimalism and closeness to plain TeX, with more difficult tasks being offloaded to Lua code, making expl3 feel somewhat out of place. ConTeXt, likewise, is Lua-focused, with users more accustomed to Lua than to TeX macros.

That's not true. Most users use the tex input language. They might kick in soem metapost (metafun) and lua, or use a mix with lua when data has to be massaged, but coding mostly text document input in a programming language like lua is counter productive and especially for more complex and rich coding also a pain (think passing arguments and evaluation in a function based language with multiple layers of nesting).

For OPTeX, we recommend using `markdown.lua` directly. I wonder if that approach might also be the most appropriate option for ConTeXt.

I don't know. It can be kind of amusing to see comments wrt context and luatex (i'm not talking of you here). For instance I remember a comparison in TB between optex being minimal (comparing format file size) while optex loads the fontloader and other stuff at runtime (which size is then ignored) and of course the macro base (i assume) will grow over time too and that all then makes the original comparisons and arguments kind of silly. The same is actually true for arguments for using latex over context ... successive past arguments get invalidated but naratives stay arround. A lot of weird and inconstent politics can enter tex related discussions and those could never really interest me, especially not when (technical) arguments make no sense and can easily be countered (but why waste time on it; it's not my duty to debunk). Anyway, in order to test this ; issue I downloaded the expl generic file and (with a change in the lua file -- different function) it could be loaded but esp the unicode part takes quite some time, likely more than users will accept in an edit-run cycle that normally takes a few seconds per run. I suppose that over time more will get loaded and that for sure will start clashing with macro package code / approaches so that might something that you need to keep an eye on: reduncancy, clashes, bloat, runtime, and whatever bothers the mkdown module. I can't comment on what gets and has to be loaded (apart fromn runtime) but i can imagine that it takes some effort from your end to make all macro packages happy. Of course, when you need some specific helpers we can provide them, Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Dear Hans, Returning to this after a couple of months: we now seem to be on track to get the Markdown package working with ConTeXt LMTX [1] and introducing ConTeXt Standalone into our continuous integration process. On Mon, Nov 11, 2024 at 01:04:08PM +0100, Hans Hagen wrote:

Anyway, in order to test this ; issue I downloaded the expl generic file and (with a change in the lua file -- different function) it could be loaded but esp the unicode part takes quite some time, likely more than users will accept in an edit-run cycle that normally takes a few seconds per run. [...]

As you noted, loading `expl3-generic.tex` in ConTeXt introduces a significant delay on each run. To mitigate this, I'm considering including the file in the ConTeXt format files -- at least within our Docker images for starters. Is there any documentation (beyond the source code) on how to do this properly, or should I just go ahead and bring up any questions here as they arise? [1]: https://github.com/Witiko/markdown/pull/557 Best, Vítek