September 2005 - dev-luatex - NTG Mailing Lists

Karel's presentation on TUG 2005
by Karel Skoupý 29 Sep '05

29 Sep '05

Hi All, reacting to Thanh's request, I put my TUG 2005 presentation on http://www.inf.ethz.ch/personal/kaskoupy/slides/FreeShapeTextFormatting.pdf I doubt that there can be anything useful understood just from those slides. I should still write a proper paper in next few days, that should be much more clear and complete, you can safely wait for that. I'll also send you the actual prototype with the instructions how to get it running, hopefully tonight. Regards, --ksk

1 0

Re: [Dev-luatex] plugin for external formatting
by Thanh Han The 25 Sep '05

25 Sep '05

Hi, this is a long thread and too many things are discussed at the same time so I will need some time to read and understand what is going on. My first thought is that some small modifications to \showlist and \showbox will help a lot. It's easy to write additional info like dimensions of each item in the list, or in case of characters the filename of a tfm with fontsize (or we may write the dimensions of each char as Hans suggested, but this is an overkill IMHO). My feeling is that we need to work out the specification and format of the `` node list'' first. In the first step, I would prefer to have only node-specific things, eg only what comes out after a box construction. I also got a similar request: to provide a primitive that writes out the content of a box and another primitive to re-construct that box back from the output. We can start with this and make further extensions later on. At the moment I cannot see clearly what is needed, but I am willing to write some extensions so that we can experiment with to see what is really needed and perhaps change what have been done. Thanh On Tue, Sep 20, 2005 at 11:58:08AM +0200, Karel Skoupy wrote: > Hi all, > > as Hans has already mentioned, my concern with luatex is to have some > interface/protocol for formatting the TeX stuff externally. > > Without going to details now, I'm interested in alternative algorithms for > formatting not only paragraphs, but the whole stream. For TUG 2005 I have > written a prototype which doesn't use any TeX code at all (it just > parasities on ADvi code for getting some metric information and showing the > results). For long time I planned to make a whole new system from scratch, > but for several reasons, that was reconsidered and Hans proposed a way > (plugin mechanism for external engine), how to cooperate with TeX, so TeX > could benefit from the new algorithm and I can concentrate on the core > stuff. > > So basically I need a stream of (character) boxes, glues, penalties, ... > (is there a simple unambiguous notion for all that?) in a preprocessed > form (I don't care about input and macro handling) plus some parameters > (standard paragraph breaking parameters and the new special ones) and I > will return a stream of fixed boxes. > > 'I' will often mean 'the engine' depending on the context :-) > > In the first stage, I won't need lua (or any changes to TeX) at all. I > plan to use \showlists for my input stream and to generate a standard > TeX input file for reading the result back. Of course, it won't be so > simple, there will be some macro programming and trickery, which will > make the whole thing complicated, fragile, unreliable, and inefficient for > real use. Therefore some hooks from the actively developed TeX will be > probably useful for making the cooperation of TeX and the external > engine smooth. It might use lua or not, we will see, in any case I would > like to keep the plugin support generic and (complete but) minimal. > > I will now list the aspects of the communications between TeX and the > engine which I have thought of so far. I will be glad if you can just think > about it for the moment and give me some feedback if you will. > > * single paragraph stuff > > I need: > (1) complete representation of all the stuff which is to be returned formatted > (2) sizes of all the objects which are involved in formatting > (3) properties which influence the formatting (breakable, discardable, ...) > > It seems that the standard output of \showlists (or \showbox) will mostly do. > (1) is fulfilled I guess (the returned input needs to be only slightly > modified to fit TeX). > > (2) is little bit tricky, because for the characters I get only an id of > the font. So I will need to know the exact reference to a real font to get > the metrics information. This can be learned by eg. \show\tenrm. But of > course it is not know in advance what fonts are used in the paragraph, so > either all fonts can be listed at the beginning -- but where to get the list of > all font definitions, and the definitions can actually change in the middle > of the paragraph -- or I can make a first pass, collect the font ids and > ask for them in the second pass. It will be bit tricky and won't be > reliable due to redefinitions (I can also change the current id using \let > and lose the old id (still used in the log), right?), so it will be OK for > experimenting but for a real version, I will need a better support from > TeX. > > (3) is implicit, right? > > * stream of paragraphs > > I can need even the whole chapter, because I want to treat > - shapes and layouts, which are relative to page and not to a particular > paragraph > - pagination, floats placement > > For the basic experimenting I can redefine \par to something like > \hfil\break\indent but it will restrict all kinds of things which can > happen between the paragraphs (in vertical mode). Of course, the whole > thing will never be compatible to TeX, because TeX expect after \par that > the last paragraph was formatted and placed on the vertical list. So it > will be responsibility of the user/macro-programmer to bear the > consequences of using the alternative mechanism. Nevertheless, the > consequences should be as small as possible. > > So for the prototyping I can redefine \par or perhaps I can store the whole > paragraphs in infinite hboxes (redefining \hsize?) or maybe I can use some > \specials for tagging, but for the production version, this will be a very > tricky part. Not so much for the engine, but mainly on the TeX side. It > should be of a great concern for people who would want to use the new > algorithms in their systems (Hans?), (after those ideas are first tested by > a prototype :-). > > * passing the parameters specific to the new algorithms > > - layouts, shapes > - maybe others, like weights for resolving paragraph contra page breaking > > This will be a new thing so I hope that there is no compatibility burden. > > * hyphenation > > It will be a lot of additional work, but I think that I should handle it > locally. There are two reasons: > > (1) the protocol for failing and getting the list with new discretionaries > (TeX's 2nd pass) for every individual paragraph would be extremely > complicated, in the end it might be more difficult than handling it locally. > > (2) TeX's hyphenation mechanism is IMHO one of the crappiest parts of TeX. > I mean the way how the (non)ligatures are screwed up for discretionaries > which are not used in the end. So if it is handled locally, it will be IMO > simpler and more correct. There are also some research results concerning > hyphenation, which are not implemented in TeX, because it would be too > complicated. > > At the first stage, I'll omit the hyphenation completelly. > > At the moment, I don't remember anything else. I'm looking forward for your > feedback. > > --ksk > _______________________________________________ > Dev-luatex mailing list > Dev-luatex(a)ntg.nl > http://www.ntg.nl/mailman/listinfo/dev-luatex >

3 3

Re: [Dev-luatex] plugin for external formatting
by Thanh Han The 23 Sep '05

23 Sep '05

On Thu, Sep 22, 2005 at 09:02:43AM +0200, Hans Hagen wrote: > Thanh Han The wrote: > > >My feeling is that we need to work out the specification and > >format of the `` node list'' first. In the first step, I > >would prefer to have only node-specific things, eg only what > >comes out after a box construction. I also got a similar > >request: to provide a primitive that writes out the content > >of a box and another primitive to re-construct that box back > >from the output. We can start with this and make further > >extensions later on. > > > >At the moment I cannot see clearly what is needed, but I am > >willing to write some extensions so that we can experiment with > >to see what is really needed and perhaps change what have been done. > > > > > i'd say ... go ahead, so that we get a picture; at least we then hav ea > starting point for karl's work err -- we need to work out how the node list will be output and input back first. I am still waiting for Karel's opinion. Thanh

1 0

pdfliteral after text/rules
by Hartmut Henkel 22 Sep '05

22 Sep '05

Hi, here ist the problem, that the "page" mode of \pdfliteral and \latelua does not by itself guarantee, that the result goes relative to the lower left corner (the PDF origin): \nopagenumbers \pdfcompresslevel=0 X\latelua{ x={};y={}; c=72/72.27/65536; x[1]=pdf.geth()*c; y[1]=pdf.getv()*c } \input tufte X\latelua page{ x[2]=pdf.geth()*c; y[2]=pdf.getv()*c } \hrule \latelua page{ tex.print(x[1], " ", y[1], " m ", x[2], " ", y[2], " l S") } \input tufte \bye The above example works as intended only, if you remove the \hrule. Reason is, that for setting a rule, a transform "cm" is done away from the origin, then the rule is placed, but there is no transforms back to the origin. The next placements are done after a transform relative to the placement point of the rule... Text is always typeset with the transform matrix "cm" set to the origin, and inside the BT...ET group the Td operator does the local transform. There seems to be currently no \pdfliteral (or \latelua) mode that guarantees positioning relative to the PDF origin. But positioning relative to the origin would be nice to have since we now have the pdf.geth/v() lua functions. Else one would have to think what was the state (text or rule?) just before using \pdfliteral page... To solve this ambiguity, i have tried the attached patch, which puts any rule into a qQ group and does "cm" transforms inside this group. So the rules are handled like text, which is also grouped (inside BT...ET). That means, that the global working reference point is (hopefully :) always (?) at the origin. With the rule.ch patch the above example works apparently ok. This also means, that when using \pdfliteral page, one gets stuff placed at the origin, and still without transform (which should be ok). This would spare to invent yet another \pdfliteral mode. No idea how general it is. And wonder, how backward-compatible? We would need to check also \pdfrefximage... Just for discussion... Regards, Hartmut P.S. Let's see how the attachment is handled. Never tried this before.

3 5

plugin for external formatting
by Karel Skoupý 21 Sep '05

21 Sep '05

Hi all, as Hans has already mentioned, my concern with luatex is to have some interface/protocol for formatting the TeX stuff externally. Without going to details now, I'm interested in alternative algorithms for formatting not only paragraphs, but the whole stream. For TUG 2005 I have written a prototype which doesn't use any TeX code at all (it just parasities on ADvi code for getting some metric information and showing the results). For long time I planned to make a whole new system from scratch, but for several reasons, that was reconsidered and Hans proposed a way (plugin mechanism for external engine), how to cooperate with TeX, so TeX could benefit from the new algorithm and I can concentrate on the core stuff. So basically I need a stream of (character) boxes, glues, penalties, ... (is there a simple unambiguous notion for all that?) in a preprocessed form (I don't care about input and macro handling) plus some parameters (standard paragraph breaking parameters and the new special ones) and I will return a stream of fixed boxes. 'I' will often mean 'the engine' depending on the context :-) In the first stage, I won't need lua (or any changes to TeX) at all. I plan to use \showlists for my input stream and to generate a standard TeX input file for reading the result back. Of course, it won't be so simple, there will be some macro programming and trickery, which will make the whole thing complicated, fragile, unreliable, and inefficient for real use. Therefore some hooks from the actively developed TeX will be probably useful for making the cooperation of TeX and the external engine smooth. It might use lua or not, we will see, in any case I would like to keep the plugin support generic and (complete but) minimal. I will now list the aspects of the communications between TeX and the engine which I have thought of so far. I will be glad if you can just think about it for the moment and give me some feedback if you will. * single paragraph stuff I need: (1) complete representation of all the stuff which is to be returned formatted (2) sizes of all the objects which are involved in formatting (3) properties which influence the formatting (breakable, discardable, ...) It seems that the standard output of \showlists (or \showbox) will mostly do. (1) is fulfilled I guess (the returned input needs to be only slightly modified to fit TeX). (2) is little bit tricky, because for the characters I get only an id of the font. So I will need to know the exact reference to a real font to get the metrics information. This can be learned by eg. \show\tenrm. But of course it is not know in advance what fonts are used in the paragraph, so either all fonts can be listed at the beginning -- but where to get the list of all font definitions, and the definitions can actually change in the middle of the paragraph -- or I can make a first pass, collect the font ids and ask for them in the second pass. It will be bit tricky and won't be reliable due to redefinitions (I can also change the current id using \let and lose the old id (still used in the log), right?), so it will be OK for experimenting but for a real version, I will need a better support from TeX. (3) is implicit, right? * stream of paragraphs I can need even the whole chapter, because I want to treat - shapes and layouts, which are relative to page and not to a particular paragraph - pagination, floats placement For the basic experimenting I can redefine \par to something like \hfil\break\indent but it will restrict all kinds of things which can happen between the paragraphs (in vertical mode). Of course, the whole thing will never be compatible to TeX, because TeX expect after \par that the last paragraph was formatted and placed on the vertical list. So it will be responsibility of the user/macro-programmer to bear the consequences of using the alternative mechanism. Nevertheless, the consequences should be as small as possible. So for the prototyping I can redefine \par or perhaps I can store the whole paragraphs in infinite hboxes (redefining \hsize?) or maybe I can use some \specials for tagging, but for the production version, this will be a very tricky part. Not so much for the engine, but mainly on the TeX side. It should be of a great concern for people who would want to use the new algorithms in their systems (Hans?), (after those ideas are first tested by a prototype :-). * passing the parameters specific to the new algorithms - layouts, shapes - maybe others, like weights for resolving paragraph contra page breaking This will be a new thing so I hope that there is no compatibility burden. * hyphenation It will be a lot of additional work, but I think that I should handle it locally. There are two reasons: (1) the protocol for failing and getting the list with new discretionaries (TeX's 2nd pass) for every individual paragraph would be extremely complicated, in the end it might be more difficult than handling it locally. (2) TeX's hyphenation mechanism is IMHO one of the crappiest parts of TeX. I mean the way how the (non)ligatures are screwed up for discretionaries which are not used in the end. So if it is handled locally, it will be IMO simpler and more correct. There are also some research results concerning hyphenation, which are not implemented in TeX, because it would be too complicated. At the first stage, I'll omit the hyphenation completelly. At the moment, I don't remember anything else. I'm looking forward for your feedback. --ksk

3 9

lua keyword { }
by Hans Hagen 18 Sep '05

18 Sep '05

Hi, So far we have \lua {...} : expanded, result of tex.print is inserted with catcodes 11/12 \lua otr {...} : delayed expansion (in otr), only pdf.print is used, tex.print is ignored currently we need to use \scantokens to process the result of tex.print in a texie way (current catcode regime). So, in addition we can have: \lua tex {...} : expanded, result of tex.print is fed into internal scantokens This saves passing args around. we also need: \luatexversion (currently 0.1.5) \luaversion (currently 5.0.2) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

2 6

FWD: Subversion Repository
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: lua Date: Tue, 13 Sep 2005 15:37:42 +0200 From: Taco Hoekwater <taco(a)elvenkind.com> To: Taco Hoekwater <taco(a)elvenkind.com> CC: Hans Hagen <pragma(a)wxs.nl>, Hartmut Henkel <hartmut_henkel(a)gmx.de> References: <4325FF95.2080808(a)wxs.nl> <Pine.LNX.4.61.0509130030060.5488(a)hahepc1.hahe> <4326075F.9030301(a)wxs.nl> <43267768.1020409(a)elvenkind.com> <43268242.5010306(a)wxs.nl> <4326878E.6070300(a)elvenkind.com> <432692BB.7070806(a)elvenkind.com> Hi, Ok, I've committed what is essentially * pdftex 1.30.3 (from sarovar) * the src directory from the lua5.1-alpha release (lua.org) * lua.ch / luastuff.c (our sources) The build system is just like pdftex, and it 'works for me'. Also, I've added Hans to the project. Main project page: https://foundry.supelec.fr/projects/luatex/ Cheers, Taco Taco Hoekwater wrote: > Hi again, > > The project is approved already :-) > > If you both register as users, then I can add you to the project. > > https://foundry.supelec.fr/ > > > Taco Hoekwater wrote: > >> >> I've requested a project at supelec. I assume Fabrice will approve >> that in a day or so, and then I will upload sources there (based >> on the "standalone" pdfetex from sarovar). > >

1 0

FWD: Re: The Other LuateX
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: LuateX Date: Thu, 15 Sep 2005 08:55:11 +0200 From: Taco Hoekwater <taco(a)elvenkind.com> To: Hans Hagen <pragma(a)wxs.nl> CC: Hartmut Henkel <hartmut_henkel(a)gmx.de> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> <432756B3.10307(a)wxs.nl> <Pine.LNX.4.61.0509142201430.4684(a)hahepc1.hahe> <432886F0.9020304(a)wxs.nl> Hans Hagen wrote: > Hartmut Henkel wrote: > >> just googled for luatex. To my surpries, many hits... >> >> http://www.luatex.com/ Unfortunate, I should have checked for that before registering the new domain. :-( >> > ha, interesting, > > so we nee to change to: > pdfluatex I much favor this solution, because it allows the continued use of the domain as well as the foundry project, and most of the other proposals were a bit too short for my taste :-) Taco

1 0

FWD: The other LuateX
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: LuateX Date: Wed, 14 Sep 2005 22:24:16 +0200 From: Hans Hagen <pragma(a)wxs.nl> To: Hartmut Henkel <hartmut_henkel(a)gmx.de> CC: Taco Hoekwater <taco(a)elvenkind.com> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> <432756B3.10307(a)wxs.nl> <Pine.LNX.4.61.0509142201430.4684(a)hahepc1.hahe> Hartmut Henkel wrote: >just googled for luatex. To my surpries, many hits... > >http://www.luatex.com/ > > ha, interesting, so we nee to change to: pdfluatex eltex (e-lua-tex) letex (lua-e-tex) lentex (lua enhanced tex -> later pdflentex) lextex (lua extended tex -> latex plextex) lunatex (i like lentex -) as working title; in the end it will be pdftex anyway Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0

FWD: \latelua
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: [macleod(a)cvrti.utah.edu: changebar support for pdflatex] Date: Wed, 14 Sep 2005 00:46:11 +0200 From: Hans Hagen <pragma(a)wxs.nl> To: Hartmut Henkel <hartmut_henkel(a)gmx.de> CC: Taco Hoekwater <taco(a)elvenkind.com> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> Hartmut Henkel wrote: >Hi Hans and Taco, > >here's a first rough try on \latelua: It just creates a node like >\pdfliteral, and the given program code is executed by lua during >shipout. Output (if any) currently goes via print into the PDF. > >If we would get access to the position cur_h, cur_v valid at the point >of lua call (Taco has this already?), we could then e. g. draw the >changebars (and other random drawings on the page between previously >marked points :-) with lua, single pass. > >All lua register changes are obviously available to "later" both \lua or >\latelua calls... > >Currently the \latelua has the same options "page" and "direct" as >\pdfliteral (because this happened by copying the code). > >Couldn't such a \latelua be removed if one would change \lua to >\immediate\lua and \lua? Or is this something different? > >Sorry, still old system, no tex.print yet and the other fancy stuff from >Taco. > > interesting i don't like the \immediate\lus thing, too texie -) maybe better than \latelua is: \lua otr {...} since there can be more variants. I also wonder ... how about pdf.print.direct("...") pdf.print.page("...") (one can always say pdf.direct = pdf.print.direct) this is more luaish, esp if we also have: pdf.h (or pdf.x) pdf.v (or pdf.y) Interesting experiment Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0