On 5/27/2019 5:52 PM, Marcel Fabian Krüger wrote:

Anyway I do not understand what you say with your last statement. How would they help solving the original problem of `#` delimiters? As far as I can tell they are rewritten into "normal" positional arguments, so I do not see how they could help. It's indeed unrelated so of no help and anyway something context specific that is nto used that often (performance wise, it started out as an experiment as usual). Anyhow, your patch deals with a very specific case and because normally the way to deal with this is via catcode settings it is not something that will end up in the engine. It deals with a # as delimiter but if these occur in a document then other cases of # will still be an issue. Making this one exception / catch is not that consistent. One way out is to not bark on # (which are special) but on the other hand this is a tex property that has been dealt with for ages in macro packages and patching the engine for that is no option (once we start along that route we end up with lots of curious patches for rare cases which is not what we have in mind).

fwiw: I have considered extending the macro handler, e.g. with more than 9 arguments which is not that hard but it makes not much sense. Macro packages have to deal with other engines too. note: As one of the complaints about luatex is that we kept adding stuff which is then voiced as equivalent for it being not stable which in turn becomes kind of boring / annoying to endure, at some point one has to decide to freeze functionality. So don't expect much additions. I'm considering a few low level optimizations and non intrusive thingies as side effect of some experiments but that will be a once-in-a-while adventure in sync with version bumps in the first digit (like 1.20, 1.30 and so). Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

On 5/27/2019 6:49 PM, Marcel Fabian Krüger wrote:

either, but I will just mention it anyway: Would it maybe be a option to provide access to the token list of macros instead? Meaning some variant of `get_macro/set_macro` which allows setting/getting the token list as returned by token.scan_toks(true, -)? This wouldn't affect any old code and not have any speciall effect for the TeX side, but would make working with macros from Lua much easier, especially if they contain too inconsistant (or unknown) catcodes to make going through a string representation a viable option. It would e.g. allow doing this `#` stuff through Lua, atomic replacements in existing macros and creating macros with more "creative" catcodes from Lua code without requiring having to go through odd `\def` constructions. Giving that the internal argument list representation is exposed through scan_toks anyway, it looks like a quite small and consistant change. Hm, not in the short term I think, but i'll keep it in mind. There's all kind of stuff going on like triggers and sentinels for arguments and ref counting and so. I also think that going too wild with lua is not a good idea. The node interface is already adding quite some fragility. The get/setmacro are actually kind of strange. One can do a lot at the tex end and if it can't be done it's probabbly also not well suited for tex. Messing with creative catcodes is no option as there is just a limited set with prescribed and often hard coded behaviour.

The whole define and expand machinery is very optimized and cannot be made much better in terms of performance, only worse and that is no option. Writing for instance that kind of stuff in lua (or adding callbacks) makes not much sense because when performance drops significantly tex basically becomes unuseable. fwiw: I've done all kind of experiments the last decade, also like the ones you suggest above, and rejected mosty because of messyness, weirdness and bordercaseness. At some point one (i can't speak for all of course) decides that there is not that much more to gain. In fact, if we remove (or bypass) some of the peculiar tex macro things one basically makes tex into something different ... some of its charm is in its weirdness ... i would have quit using it already long ago if that weren't the case. Of course I play with possible new things but it has to prove itself in usage and looking-kind-of-normaal (I admit that in luatex some things were added that don't honor that principle but removing that now is no option I think, although I have no problems kicking out strange stuff in the follow up that I experiment with in context.) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------