Hans, after your explanation I'm actually curious now about details, but my knowledge is too limited now (maybe reading source codes would be better? Worse?). I do a bit of C, but I don't know about Pascal at all and I'm not sure where to start from in order to understand TeX better. Well, at least I know why that feature isn't supported. I'll be thankful for any references. Thank you very much. Jairo :) El lun., 29 de jun. de 2020 a la(s) 02:28, Hans Hagen (j.hagen@xs4all.nl) escribió:

On 6/28/2020 10:48 PM, Jairo A. del Rio wrote:

...

I've read the following is not possible in TeX

\def#1\macro{blabla#1}

where arguments come before. The only partial exceptions are commands like \atop or \over, which are in fact primitives. Is there a way to do this in ConTeXt?

Could it be a feature request for LuaMetaTeX? I've seen Hans experimenting a lot with new primitives and new possibilities for arguments, like #0 and co., so I ask in case it's not too nonsensical to propose it. Regards Every \foo will be looked up, so by the time \macro in:

bla bla {\bf xxx}\macro{xxx}

is seen, the {\bf xxx} is already passed and processed. TeX never looks back, which actually would make for a pretty complex multipass parsing and expansion management (forward control and backward: \expandafter would then also have an \expandbefore companion). Even in the simple case: should it keep track of quantities done (grouped, single token, box, etc.) and then in retrospect see it as #1 (them being nodes by now and not tokens)?

So, why in math but not in text? The \atop and \over (those are basically all the same command but with a different treatment afterwards) are an exception: (1) tex knows that is is in math mode, and in math mode the { } are not really arguments but defines some stuff handled together. Much processing (not all) is delayed to a second pass, so {1}\over{2} internally becomes \over{1}{2} and even that is kind of tricky because there are math styles involved (which makes for some hard coded behaviour that in the perspective of luametatex i try to get more grip on). Now, in order to handle this one (!) exception to lookahead parsing, special tracking happens in math mode, the previous math grouped stuff is registered and adapted to the \over when seen, otherwise it stays as is. This exception also maked the code somewhat messier because there are several spots where it has to be dealt with (also think of saving and restoring states). Just imagine that there were more such commands. Believe me, you really don't want to know the details.

So the answer is "Can't be done without overhauling the whole concept".

Now, I know that the narrative is that context is a moving target (contrary to other macro packages that don't / can't change because users / publishers expect them to behave the same forever) so one can argue that for context we can follow a drastic different route, but even then, we can't shoot ourselves in the foot too often. I know that some people (read: Alan) love these {{}\foo{}} syntaxes but live would have been simpler if even \over has not be in there with prefix notation (there is a reason why macro package have \frac like variants).

Hans

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Sorry, that was a bit too much oversimplification:

On 29 Jun 2020, at 12:36, Taco Hoekwater wrote:

In math mode, there *is* an expression state maintained, and that is why \over and \atop work.

Actually (still oversimplifying), the found commands are saved in a temporary list, that is then reprocessed once math mode has ended. And at that stage, an expression tree is built up. Best wishes, Taco

I'll take a look at it. Thank you a lot, Taco. Regards, Jairo :) El lun., 29 de jun. de 2020 a la(s) 05:36, Taco Hoekwater ( taco@elvenkind.com) escribió:

Hi,

Oversimplifying:

TeX (and derivatives like luatex) read the input file as a list of operators with optional suffix arguments.

In normal (text) mode, there is no expression state maintained except inside the handling of such optional arguments.

It follows that, if there is a top-level text input like

a\bold{a}

TeX treats this as (assuming ’simple’ macro definitions):

command a (no arguments) command \bold (a macro with one one braced argument)

And by the time “\bold" is seen, the “a” command has already been processed completely.

In math mode, there *is* an expression state maintained, and that is why \over and \atop work.

For further information and actual details, you can just look at the luatex source (it is in C):

https://github.com/TeX-Live/luatex/blob/trunk/source/texk/web2c/luatexdir/te...

The function

void main_control(void)

is the one that does all the heavy lifting (using a dispatch table).

Taco

...

On 29 Jun 2020, at 12:15, Jairo A. del Rio wrote:

Hans, after your explanation I'm actually curious now about details, but my knowledge is too limited now (maybe reading source codes would be better? Worse?). I do a bit of C, but I don't know about Pascal at all and I'm not sure where to start from in order to understand TeX better. Well, at least I know why that feature isn't supported. I'll be thankful for any references. Thank you very much.

Jairo :)

El lun., 29 de jun. de 2020 a la(s) 02:28, Hans Hagen (j.hagen@xs4all.nl) escribió: On 6/28/2020 10:48 PM, Jairo A. del Rio wrote:

...

I've read the following is not possible in TeX

\def#1\macro{blabla#1}

where arguments come before. The only partial exceptions are commands like \atop or \over, which are in fact primitives. Is there a way to do this in ConTeXt?

Could it be a feature request for LuaMetaTeX? I've seen Hans experimenting a lot with new primitives and new possibilities for arguments, like #0 and co., so I ask in case it's not too nonsensical to propose it. Regards Every \foo will be looked up, so by the time \macro in:

bla bla {\bf xxx}\macro{xxx}

is seen, the {\bf xxx} is already passed and processed. TeX never looks back, which actually would make for a pretty complex multipass parsing and expansion management (forward control and backward: \expandafter would then also have an \expandbefore companion). Even in the simple case: should it keep track of quantities done (grouped, single token, box, etc.) and then in retrospect see it as #1 (them being nodes by now and not tokens)?

So, why in math but not in text? The \atop and \over (those are basically all the same command but with a different treatment afterwards) are an exception: (1) tex knows that is is in math mode, and in math mode the { } are not really arguments but defines some stuff handled together. Much processing (not all) is delayed to a second pass, so {1}\over{2} internally becomes \over{1}{2} and even that is kind of tricky because there are math styles involved (which makes for some hard coded behaviour that in the perspective of luametatex i try to get more grip on). Now, in order to handle this one (!) exception to lookahead parsing, special tracking happens in math mode, the previous math grouped stuff is registered and adapted to the \over when seen, otherwise it stays as is. This exception also maked the code somewhat messier because there are several spots where it has to be dealt with (also think of saving and restoring states). Just imagine that there were more such commands. Believe me, you really don't want to know the details.

So the answer is "Can't be done without overhauling the whole concept".

Now, I know that the narrative is that context is a moving target (contrary to other macro packages that don't / can't change because users / publishers expect them to behave the same forever) so one can argue that for context we can follow a drastic different route, but even then, we can't shoot ourselves in the foot too often. I know that some people (read: Alan) love these {{}\foo{}} syntaxes but live would have been simpler if even \over has not be in there with prefix notation (there is a reason why macro package have \frac like variants).

Hans

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

___________________________________________________________________________________

...

If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net

___________________________________________________________________________________

Taco Hoekwater Elvenkind BV

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net

___________________________________________________________________________________

Wouldn't it be slow for larger documents? I will try anyway. By the way, I've seen Lua(La)TeX users recurring to callbacks (process input buffer) to make string replacements and I've done so myself sometimes. Does ConTeXt do preprocessing the same way or is a better alternative possible? Thank you very much. Jairo :) El lun., 29 de jun. de 2020 a la(s) 02:50, luigi scarso ( luigi.scarso@gmail.com) escribió:

On Sun, Jun 28, 2020 at 10:48 PM Jairo A. del Rio wrote:

...

I've read the following is not possible in TeX

\def#1\macro{blabla#1}

where arguments come before. The only partial exceptions are commands like \atop or \over, which are in fact primitives. Is there a way to do this in ConTeXt?

you can build a lpeg and parse the your "extended tex" source, obtaining a valid tex source.

-- luigi

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net

___________________________________________________________________________________

Oh, I was just asking about luatexbase.add_to_callback("process_input_buffer", blabla, "blabla" ) and luatexbase.remove_from_callback("process_input_buffer", "blabla" ) LuaLaTeX users do that way and I want to be sure it won't interfere with ConTeXt way of doing things so I can play with the suggestion made by Luigi. Regards, Jairo :) El lun., 29 de jun. de 2020 a la(s) 08:27, Hans Hagen (j.hagen@xs4all.nl) escribió:

On 6/29/2020 12:21 PM, Jairo A. del Rio wrote:

...

Wouldn't it be slow for larger documents? I will try anyway. By the way, I've seen Lua(La)TeX users recurring to callbacks (process input buffer) to make string replacements and I've done so myself sometimes. Does ConTeXt do preprocessing the same way or is a better alternative possible? Thank you very much. You mean "Does context use callbacks?" You can bet on that. It is how we came to implementing callbacks (over a decade ago now, Taco and I spend quite some time exploring all these things in the process of luatex dev ... fond memories and so).

I don't know what latex does but context doesn't do much input processing apart from some unicode related juggling. There are plenty of ways to hook in additional processing but there are multiple ways to solve problems. Most of these mechanism in context are already pretty old and hardly change.

Hans

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net

___________________________________________________________________________________