- dev-luatex - NTG Mailing Lists

plugin for external formatting
by Karel Skoupý 21 Sep '05

21 Sep '05

Hi all, as Hans has already mentioned, my concern with luatex is to have some interface/protocol for formatting the TeX stuff externally. Without going to details now, I'm interested in alternative algorithms for formatting not only paragraphs, but the whole stream. For TUG 2005 I have written a prototype which doesn't use any TeX code at all (it just parasities on ADvi code for getting some metric information and showing the results). For long time I planned to make a whole new system from scratch, but for several reasons, that was reconsidered and Hans proposed a way (plugin mechanism for external engine), how to cooperate with TeX, so TeX could benefit from the new algorithm and I can concentrate on the core stuff. So basically I need a stream of (character) boxes, glues, penalties, ... (is there a simple unambiguous notion for all that?) in a preprocessed form (I don't care about input and macro handling) plus some parameters (standard paragraph breaking parameters and the new special ones) and I will return a stream of fixed boxes. 'I' will often mean 'the engine' depending on the context :-) In the first stage, I won't need lua (or any changes to TeX) at all. I plan to use \showlists for my input stream and to generate a standard TeX input file for reading the result back. Of course, it won't be so simple, there will be some macro programming and trickery, which will make the whole thing complicated, fragile, unreliable, and inefficient for real use. Therefore some hooks from the actively developed TeX will be probably useful for making the cooperation of TeX and the external engine smooth. It might use lua or not, we will see, in any case I would like to keep the plugin support generic and (complete but) minimal. I will now list the aspects of the communications between TeX and the engine which I have thought of so far. I will be glad if you can just think about it for the moment and give me some feedback if you will. * single paragraph stuff I need: (1) complete representation of all the stuff which is to be returned formatted (2) sizes of all the objects which are involved in formatting (3) properties which influence the formatting (breakable, discardable, ...) It seems that the standard output of \showlists (or \showbox) will mostly do. (1) is fulfilled I guess (the returned input needs to be only slightly modified to fit TeX). (2) is little bit tricky, because for the characters I get only an id of the font. So I will need to know the exact reference to a real font to get the metrics information. This can be learned by eg. \show\tenrm. But of course it is not know in advance what fonts are used in the paragraph, so either all fonts can be listed at the beginning -- but where to get the list of all font definitions, and the definitions can actually change in the middle of the paragraph -- or I can make a first pass, collect the font ids and ask for them in the second pass. It will be bit tricky and won't be reliable due to redefinitions (I can also change the current id using \let and lose the old id (still used in the log), right?), so it will be OK for experimenting but for a real version, I will need a better support from TeX. (3) is implicit, right? * stream of paragraphs I can need even the whole chapter, because I want to treat - shapes and layouts, which are relative to page and not to a particular paragraph - pagination, floats placement For the basic experimenting I can redefine \par to something like \hfil\break\indent but it will restrict all kinds of things which can happen between the paragraphs (in vertical mode). Of course, the whole thing will never be compatible to TeX, because TeX expect after \par that the last paragraph was formatted and placed on the vertical list. So it will be responsibility of the user/macro-programmer to bear the consequences of using the alternative mechanism. Nevertheless, the consequences should be as small as possible. So for the prototyping I can redefine \par or perhaps I can store the whole paragraphs in infinite hboxes (redefining \hsize?) or maybe I can use some \specials for tagging, but for the production version, this will be a very tricky part. Not so much for the engine, but mainly on the TeX side. It should be of a great concern for people who would want to use the new algorithms in their systems (Hans?), (after those ideas are first tested by a prototype :-). * passing the parameters specific to the new algorithms - layouts, shapes - maybe others, like weights for resolving paragraph contra page breaking This will be a new thing so I hope that there is no compatibility burden. * hyphenation It will be a lot of additional work, but I think that I should handle it locally. There are two reasons: (1) the protocol for failing and getting the list with new discretionaries (TeX's 2nd pass) for every individual paragraph would be extremely complicated, in the end it might be more difficult than handling it locally. (2) TeX's hyphenation mechanism is IMHO one of the crappiest parts of TeX. I mean the way how the (non)ligatures are screwed up for discretionaries which are not used in the end. So if it is handled locally, it will be IMO simpler and more correct. There are also some research results concerning hyphenation, which are not implemented in TeX, because it would be too complicated. At the first stage, I'll omit the hyphenation completelly. At the moment, I don't remember anything else. I'm looking forward for your feedback. --ksk

3 9

lua keyword { }
by Hans Hagen 18 Sep '05

18 Sep '05

Hi, So far we have \lua {...} : expanded, result of tex.print is inserted with catcodes 11/12 \lua otr {...} : delayed expansion (in otr), only pdf.print is used, tex.print is ignored currently we need to use \scantokens to process the result of tex.print in a texie way (current catcode regime). So, in addition we can have: \lua tex {...} : expanded, result of tex.print is fed into internal scantokens This saves passing args around. we also need: \luatexversion (currently 0.1.5) \luaversion (currently 5.0.2) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

2 6

FWD: Subversion Repository
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: lua Date: Tue, 13 Sep 2005 15:37:42 +0200 From: Taco Hoekwater <taco(a)elvenkind.com> To: Taco Hoekwater <taco(a)elvenkind.com> CC: Hans Hagen <pragma(a)wxs.nl>, Hartmut Henkel <hartmut_henkel(a)gmx.de> References: <4325FF95.2080808(a)wxs.nl> <Pine.LNX.4.61.0509130030060.5488(a)hahepc1.hahe> <4326075F.9030301(a)wxs.nl> <43267768.1020409(a)elvenkind.com> <43268242.5010306(a)wxs.nl> <4326878E.6070300(a)elvenkind.com> <432692BB.7070806(a)elvenkind.com> Hi, Ok, I've committed what is essentially * pdftex 1.30.3 (from sarovar) * the src directory from the lua5.1-alpha release (lua.org) * lua.ch / luastuff.c (our sources) The build system is just like pdftex, and it 'works for me'. Also, I've added Hans to the project. Main project page: https://foundry.supelec.fr/projects/luatex/ Cheers, Taco Taco Hoekwater wrote: > Hi again, > > The project is approved already :-) > > If you both register as users, then I can add you to the project. > > https://foundry.supelec.fr/ > > > Taco Hoekwater wrote: > >> >> I've requested a project at supelec. I assume Fabrice will approve >> that in a day or so, and then I will upload sources there (based >> on the "standalone" pdfetex from sarovar). > >

1 0

FWD: Re: The Other LuateX
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: LuateX Date: Thu, 15 Sep 2005 08:55:11 +0200 From: Taco Hoekwater <taco(a)elvenkind.com> To: Hans Hagen <pragma(a)wxs.nl> CC: Hartmut Henkel <hartmut_henkel(a)gmx.de> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> <432756B3.10307(a)wxs.nl> <Pine.LNX.4.61.0509142201430.4684(a)hahepc1.hahe> <432886F0.9020304(a)wxs.nl> Hans Hagen wrote: > Hartmut Henkel wrote: > >> just googled for luatex. To my surpries, many hits... >> >> http://www.luatex.com/ Unfortunate, I should have checked for that before registering the new domain. :-( >> > ha, interesting, > > so we nee to change to: > pdfluatex I much favor this solution, because it allows the continued use of the domain as well as the foundry project, and most of the other proposals were a bit too short for my taste :-) Taco

1 0

FWD: The other LuateX
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: LuateX Date: Wed, 14 Sep 2005 22:24:16 +0200 From: Hans Hagen <pragma(a)wxs.nl> To: Hartmut Henkel <hartmut_henkel(a)gmx.de> CC: Taco Hoekwater <taco(a)elvenkind.com> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> <432756B3.10307(a)wxs.nl> <Pine.LNX.4.61.0509142201430.4684(a)hahepc1.hahe> Hartmut Henkel wrote: >just googled for luatex. To my surpries, many hits... > >http://www.luatex.com/ > > ha, interesting, so we nee to change to: pdfluatex eltex (e-lua-tex) letex (lua-e-tex) lentex (lua enhanced tex -> later pdflentex) lextex (lua extended tex -> latex plextex) lunatex (i like lentex -) as working title; in the end it will be pdftex anyway Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0

FWD: \latelua
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: [macleod(a)cvrti.utah.edu: changebar support for pdflatex] Date: Wed, 14 Sep 2005 00:46:11 +0200 From: Hans Hagen <pragma(a)wxs.nl> To: Hartmut Henkel <hartmut_henkel(a)gmx.de> CC: Taco Hoekwater <taco(a)elvenkind.com> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> Hartmut Henkel wrote: >Hi Hans and Taco, > >here's a first rough try on \latelua: It just creates a node like >\pdfliteral, and the given program code is executed by lua during >shipout. Output (if any) currently goes via print into the PDF. > >If we would get access to the position cur_h, cur_v valid at the point >of lua call (Taco has this already?), we could then e. g. draw the >changebars (and other random drawings on the page between previously >marked points :-) with lua, single pass. > >All lua register changes are obviously available to "later" both \lua or >\latelua calls... > >Currently the \latelua has the same options "page" and "direct" as >\pdfliteral (because this happened by copying the code). > >Couldn't such a \latelua be removed if one would change \lua to >\immediate\lua and \lua? Or is this something different? > >Sorry, still old system, no tex.print yet and the other fancy stuff from >Taco. > > interesting i don't like the \immediate\lus thing, too texie -) maybe better than \latelua is: \lua otr {...} since there can be more variants. I also wonder ... how about pdf.print.direct("...") pdf.print.page("...") (one can always say pdf.direct = pdf.print.direct) this is more luaish, esp if we also have: pdf.h (or pdf.x) pdf.v (or pdf.y) Interesting experiment Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0

Re: [Dev-luatex] pick up the thread
by Thanh Han The 15 Sep '05

15 Sep '05

Hi, very interesting and promissing indeed. Great idea! If this happened sooner, a lot of effort on extending pdftex could have been saved. Thanh On Wed, Aug 31, 2005 at 11:48:59AM +0200, Hans Hagen wrote: > Hi, > > Since this is the first mail to the list ... i attach the presentation > done at tug 2005 since it summarizes the state of the project as well as > some demo's at that time. There may be errors in it, but beter have this > than a bunch of dutch/english mails. > > Hans > > > ----------------------------------------------------------------- > Hans Hagen | PRAGMA ADE > Ridderstraat 27 | 8061 GH Hasselt | The Netherlands > tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com > | www.pragma-pod.nl > ----------------------------------------------------------------- > > > _______________________________________________ > Dev-luatex mailing list > Dev-luatex(a)ntg.nl > http://www.ntg.nl/mailman/listinfo/dev-luatex

2 1

next step
by Hans Hagen 31 Aug '05

31 Aug '05

Hi, We need to consider the next steps. For instance: - in addition to registers, also support lccode/uccode/etc access - provide a \luaesc primitive that escapes characters, which makes it possible to say for instance \lua{str = "\luaesc{#1}"} with #1 containing " - provide access to the hash, i.e. tex.hash["whatever"] = "some text" as well as str = tex.hash["whatever"] we need to figure out a clean way to use variables like pagegoal, maybe just tex.pagegoal - consider the luapdftex variant to use omega like registers instead of etex sparse arrays just a start -) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0

pick up the thread
by Hans Hagen 31 Aug '05

31 Aug '05

Hi, Since this is the first mail to the list ... i attach the presentation done at tug 2005 since it summarizes the state of the project as well as some demo's at that time. There may be errors in it, but beter have this than a bunch of dutch/english mails. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0