A few idle questions...

David Kastrup

7 May 2008 7 May '08

2:03 p.m.

Hi, XML processing is sort of the rage of the day, and ConTeXt also has some support for it. What I have been wondering is whether luaexpat is on the list of libraries where inclusion into LuaTeX would be considered. Either directly, or as a configuration option when compiling. For processing pure XML files or fragments, this would seem like interesting infrastructure. Another question: I found inclusion of the the Coco patches quite interesting. An application that came immediately to my mind was having a control flow written in Lua that feeds TeX code into the TeX interpreter and works with the result from executing these TeX instructions, by yielding a Lua coroutine until the executed TeX code finally resumes it. Was that the basic intention? Are there any practical experiences with Coco in LuaTeX already? All the best, David -- David Kastrup

Show replies by date

Hans Hagen

7 May 7 May

2:16 p.m.

David Kastrup wrote:

...

Hi,

XML processing is sort of the rage of the day, and ConTeXt also has some support for it. What I have been wondering is whether luaexpat is on the list of libraries where inclusion into LuaTeX would be considered. Either directly, or as a configuration option when compiling.

additional parsing libraries are not on the agenda; at some point there may be a way to load libraries at runtime (we have decided to postpone this for a while); maintaining support for such libraries is not our primary task either

...

For processing pure XML files or fragments, this would seem like interesting infrastructure.

there are too many approached to this, and in order to avoid wrong choices and endles discussions, the way to deal with this is either to use lua or to use a library; since luatex is open source users can add such things locally (after all, this is what open source is about);

...

Another question: I found inclusion of the the Coco patches quite

these extensions were added after some discussions with James Quirk who want to experiment with it

...

interesting. An application that came immediately to my mind was having a control flow written in Lua that feeds TeX code into the TeX interpreter and works with the result from executing these TeX instructions, by yielding a Lua coroutine until the executed TeX code finally resumes it. Was that the basic intention? Are there any practical experiences with Coco in LuaTeX already?

the short term objectives are to open up tex and in the process many 'components' are rewritten, for example the components of paragraph building already have been isolated; a long term objective (already mentioned in roadmaps) is to use lua as the language that glues the various components and stages together; at that point on emay decide to use coroutines but that's not the only solution (i.e. we will not focus on one model) but this major overhaul will not happen soon (> 2010) because first the code base has to be converte to cweb; also, at some point we need to guarantee stability, so such paradism shifts will not happen in luatex versions 1.* but in 2.* Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Jonathan Sauer

2:18 p.m.

Hello,

...

Are there any practical experiences with Coco in LuaTeX already?

Even though I would not call them "practical": I once used coroutines in the file reading callback to convert a text file to TeX using coroutine.yield to return each converted line of text as well as a short preamble and a postamble. I posted the example to the LuaTeX wiki: <http://luatex.bluwiki.com/go/Typeset_non-TeX_files_by_converting_them_u sing_Lua_code> It works like a charm (even though the example is not very useful). I was also thinking of implementing a LaTeX environment using a coroutine, so that there is only a single function that yields after \begin and is resumed by \end. That way, the \end code can access local variables of \begin. Example: \directlua0{ function env() % \begin: local answer = 42 tex.sprint("Beginning the environment") coroutine.yield() % \end: tex.sprint("Ending the environment with ", tostring(answer)) end } \def\env{% \directlua0{currEnv = env()}% } \def\endenv{% \directlua0{coroutine.resume(currEnv) currEnv = nil}% } \begin{env} ... \end{env} This results in (or at least should result in): Beginning the environment ... Ending the environment with 42

...

All the best, David

Jonathan

David Kastrup

2:29 p.m.

"Jonathan Sauer" writes:

...

Hello,

...
Are there any practical experiences with Coco in LuaTeX already?

Even though I would not call them "practical":

I once used coroutines in the file reading callback to convert a text file to TeX using coroutine.yield to return each converted line of text as well as a short preamble and a postamble. I posted the example to the LuaTeX wiki:

<http://luatex.bluwiki.com/go/Typeset_non-TeX_files_by_converting_them_u sing_Lua_code>

Yes, that is the kind of thing I had been thinking of. Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming.

...

I was also thinking of implementing a LaTeX environment using a coroutine,

Oh, is anybody actually using LuaTeX with LaTeX? Any guidelines about "what-to-do-to-make-it-run"? -- David Kastrup

Hans Hagen

2:41 p.m.

David Kastrup wrote:

...

Yes, that is the kind of thing I had been thinking of. Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming.

you can already use a callback to collect material that is added to the vertical lists Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

David Kastrup

2:50 p.m.

Hans Hagen writes:

...

David Kastrup wrote:

...
Yes, that is the kind of thing I had been thinking of. Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming.

you can already use a callback to collect material that is added to the vertical lists

I don't understand. What callback, and where is the material supposed to originate? What I was talking about is to let the normal TeX typesetting process come up with more material. Basically, not having to leave the current Lua context/variables/control structures for the sake of getting more material in the way "normal" for an output routine. -- David Kastrup

Hans Hagen

2:57 p.m.

David Kastrup wrote:

...

Hans Hagen writes:

...
David Kastrup wrote:

...
Yes, that is the kind of thing I had been thinking of. Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming. you can already use a callback to collect material that is added to the vertical lists

I don't understand. What callback, and where is the material supposed to originate?

vpack_filter buildpage_filter at that point you can save the list passed to the function and return nothing ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

David Kastrup

3:05 p.m.

Hans Hagen writes:

...

David Kastrup wrote:

...
Hans Hagen writes:

...
David Kastrup wrote:

...
Yes, that is the kind of thing I had been thinking of. Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming. you can already use a callback to collect material that is added to the vertical lists

I don't understand. What callback, and where is the material supposed to originate?

vpack_filter buildpage_filter

at that point you can save the list passed to the function and return nothing

I am afraid that is not useful for what I had in mind, since what I had in mind requires actual typesetting, so I need to inject material into TeX (unhboxing into vertical boxes, for example) and take a look at the results. I can reinject material into TeX while being caught in the output routine, but the callback would not seem like a suitable context for that. Namely: I can't finish the job using just Lua. I suppose I have to do my own experiments here, I just wanted to hear about existing experiences and/or approaches. -- David Kastrup

Hans Hagen

3:27 p.m.

David Kastrup wrote:

...

Hans Hagen writes:

...
David Kastrup wrote:

...
Hans Hagen writes:

...
David Kastrup wrote:

...
Yes, that is the kind of thing I had been thinking of. Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming. you can already use a callback to collect material that is added to the vertical lists I don't understand. What callback, and where is the material supposed to originate? vpack_filter buildpage_filter

at that point you can save the list passed to the function and return nothing

I am afraid that is not useful for what I had in mind, since what I had in mind requires actual typesetting, so I need to inject material into TeX (unhboxing into vertical boxes, for example) and take a look at the results. I can reinject material into TeX while being caught in the output routine, but the callback would not seem like a suitable context for that. Namely: I can't finish the job using just Lua.

I suppose I have to do my own experiments here, I just wanted to hear about existing experiences and/or approaches.

as a variant on coroutines you can just print to tex things like tex.sprint("\\directlua{nextstage()}" intermixed with other tex code this works ok but is not always beautiful but for the moment it is the only way to go ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Jonathan Sauer

3 p.m.

Hello,

...

Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming.

The "buildpage_filter" seems to be able to do this. Untested: local function cocoroutine(head, extrainfo) while not_done do -- Do something with the new material ... -- Add the modified material to the main vertical list, then -- get more material (this is effectively the function -- "get_more_main_vertical_list_material_from_TeX") -- false: Discard the material -- true: Use the original material -- new head: Use this instead of the original material head, extrainfo = coroutine.yield(true | false | new head) end end callback.register("buildpage_filter", coroutine.wrap(cocoroutine))

...

...
I was also thinking of implementing a LaTeX environment using a coroutine,

Oh, is anybody actually using LuaTeX with LaTeX? Any guidelines about "what-to-do-to-make-it-run"?

I'm sorry, I wasn't very precise in my last mail. I was talking about LaTeX-*like* environments, not actually LaTeX itself. I never tried using LaTeX with LuaTeX. Jonathan

David Kastrup

3:12 p.m.

"Jonathan Sauer" writes:

...

Hello,

...
Another is output routine programming: a function get_more_main_vertical_list_material_from_TeX would be quite handy in some applications, and yielding to TeX until the next pass would seem to make for some very straightforward and understandable programming.

The "buildpage_filter" seems to be able to do this. Untested:

[...] I'll have to see eventually whether injecting TeX code here is workable. If it is, it might be actually preferable to using the output routine with magic penalties and all that stuff that causes lots of ugly side effects in the main vertical list and the marks and box 255. Thanks for the suggestion. -- David Kastrup

Jonathan Sauer

3:19 p.m.

Hello,

...

...
The "buildpage_filter" seems to be able to do this. Untested:

I'll have to see eventually whether injecting TeX code here is workable.

If you are talking about tex.print, that won't work, at least not right now, since tex.print currently does not work in callbacks (see the discussion "process_input_buffer-callback and tex.sprint" starting yesterday). As an alternative, you could use the "token_filter" callback to return the TeX code to be injected in tokenized form when TeX asks for it. The Context MKIV manual has some examples on how to use tokens: http://pragma-ade.com/general/manuals/mk.pdf

...

David Kastrup

Jonathan

Hans Hagen

4:06 p.m.

Jonathan Sauer wrote:

...

Hello,

...
...
The "buildpage_filter" seems to be able to do this. Untested: I'll have to see eventually whether injecting TeX code here is workable.

If you are talking about tex.print, that won't work, at least not right now, since tex.print currently does not work in callbacks (see the discussion "process_input_buffer-callback and tex.sprint" starting yesterday).

sometimes you can use teh secodn call to the callback to finish actions, i.e. if a macro somehow triggers the callback, then in this macro you can do tex stuff and then again trigger the callback Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

5996

Age (days ago)

5996

Last active (days ago)

List overview

Download

12 comments

3 participants

participants (3)

David Kastrup
Hans Hagen
Jonathan Sauer