Re: [Dev-luatex] Extension language integration of LuaTeX and LilyPond

Taco Hoekwater writes:

Hi David,

On May 7, 2013, at 1:13 PM, David Kastrup wrote:

...

I've added some rough sketches at the end of the article that should make clear why this can't be done in formats alone but will require primitive support as well if things are supposed to turn out nicely.

My brain is wired such that code in the scheme language never clarifies anything, I fear.

Well, one can at least look at its input, output, and the syntax and workings of individual syntactic elements not yet wrapped into any parentheses. The implementation detail is a walk through LilyPond internals to give an impression how that apparent magic is wired. Of course, the wiring for Lua/LuaTeX would be completely different, but that it's possible to peek at how GUILE/LilyPond manages to do the raw work before coating it in parens and at least take away the impression that it is possible to do.

Nevertheless, I picked up something interesting from your postscript, and that is the tex:get_dimen() construct. Ignoring the catcode issues for the moment, I feel that the main reason why it is so hard currently to interface TeX -> Lua at the moment is the near-uselessness of the 'token' library. Extending that library (and its embedding) in such a way that it can reliably fetch various 'units' like dimensions and skips and integers from the input (either file or tokenlist), with or without expansion, would help a lot, IMO.

One feature of good programming language and interface design in my opinion is that you are finished after writing down a naïve sketch. Without then having an additional implementation phase where you are just fighting technicalities. So if a naïve sketch would include something like "get a dimension", that's what a programming interface should provide. A programmer wants to get _his_ work done, not that of the platform. In LilyPond, the point of functions like "define-music-function" is that you have mechanisms to write stuff in Scheme that can get at arguments and values just in the same way as LilyPond itself does. For one thing, it means that quite a bit of LilyPond functionality can actually be implemented in Scheme, and thus can be redefined and/or rewritten without needing to recompile. For another, Scheme is a user-level programming language. It does its own garbage collection and memory management, checks its types, and is reasonably expressive. I've encountered people without programming background who started working with LilyPond or Emacs and reached impressive mastery of them without ever getting the hang of low-level languages like C. Actually, one of such people is a major financial backer of my ongoing work on LilyPond since working on beating consistency into a programming model that was a loose connection of independent hacks at one point of time was what empowered increasingly effective work of his. That's what the "It is whole now as if it had never been broken." quote at the start of the article was about.

The Lua -> TeX route is better covered at the moment, take e.g. ConTeXt's CLD files and Patrick's typesetting engine that is almost completely lua-based.

But Context's CLD files are not part of the LuaTeX mindframe/universe to any serious degree. You never learn what they are by reading the LuaTeX manual. So their ways of doing work are Context's ways of doing work, not LuaTeX's ways of doing work. The LuaTeX manual at the current point of time is as relevant for resolving the question "why would I _want_ to use LuaTeX?" as a proof that you can use it for implementing some particular class of Turing machine. It shows more abstract capabilities rather than actual application programming strategies. Now it is nice that somewhere there are also answers to the question "why would I _want_ to use Context?". But treating them as separately important from the question "why would I want to use LuaTeX?" might expand the number of people interested in LuaTeX. -- David Kastrup