- dev-luatex - NTG Mailing Lists

[Fwd: LuaTeX]
by Taco Hoekwater 05 May '06

05 May '06

Hi, Comments are welcome before I reply. Personally I doubt much indirection is feasible, though. Taco -------- Original Message -------- Subject: LuaTeX Date: Sun, 30 Apr 2006 08:52:03 -0700 (PDT) From: James Quirk <jjq(a)galcit.caltech.edu> To: Taco Hoekwater <taco(a)elvenkind.com> Dear Taco, I understand from the LuaTeX FAQ that third-party feedback is currently not encouraged. However, I have one observation that I hope you might consider. Specifically, I would have thought it possible, and desirable, to introduce a lightweight intermediate layer to broker communications between TeX and Lua. The situation would be analagous to converting between graphics formats. Rather than writing N^2 conversion programs (a2b, a2c, b2a, b2c, c2a, c2b etc.) it is possible to get away with the 2N conversion programs (a2xyz, b2xyz, c2xyz, xyz2a, xyz2b, xyz2c etc.). The motivation being that when a new format comes along one need only construct xyz2new and new2xyz to be able to convert to any one of the other formats. So instead of having tex2lua and lua2tex communications one would have {tex,lua}2broker and broker2{tex,lua}. That way it would be be straightforward for a third-party to add their own favourite scripting language, as most of the machinations live with the broker and do not change with the choice of embedded language (at least not for languages implemented in C, which covers perl python, ruby etc.). I realise, from arguments of portablity, that you might not want to have a situation where there are 57 varieties of embedded interpreter. But the broker-design would provide a springboard for whatever language developments might occur down the road without needing to be rewritten. As for the cost of the overhead, it would not be great and with the advent of multicore processors I'm not sure it is something to worry about. Afterall, much of TeX's programming clunkiness stems from decisions that were too focussed on the machine's of the day; a mistake that crops up time and time again. In any event. I look forward to trying out LuaTeX when it is made publicly available. All the best with your project, James Quirk

2 1

new parbreaking algorthm prototype
by Karel Skoupý 03 Oct '05

03 Oct '05

Dear all, I'm sending the China version of my free-shape text breaker in case you wanna see what it does. If you really wanna compile and run it, then you need to install 2 other packages, but read till the end before you do it. (1) Objective Caml: http://caml.inria.fr/ It's a standard package in linux distros (at least in Debian), otherwise you can take binaries from the Download section at that site (2) ADvi: http://pauillac.inria.fr/advi/ My prototype links with the advi modules (v 1.6.0). Follow the download and build instructions at the site. (3) Dyntyp: dyntyp-src-050823-1121.tar.gz attached This is my prototype The actual algorithm is inside xbreak.ml. There is also break.ml which contains earlier version of the algorithm which doesn't really work. In order to build it, you need to install (1) and to untar and build (2). Then untar my package (3), preferably with the following directory layout: <SOME DIRECTORY> +--adk | +--advi-1.6.0 | +--camlimages-2.2 +--<SOME DIRECTORY> (dyntyp, but I should think of something better) +--src because (2) is referenced from (3) in Makefile.config and it expects adk in ../../adk. If you need different structure, change Makefile.config in src. Oops, I think, advi doesn't work on Windows, so the whole prototype is only for linux (unix) at the moment (but it will change soon). OK, so inside src (of (3)), do make, it should create dyntyp.opt. The shellscript show.sh runs a demo. Well, linking with ADvi was a quick hack for getting something visible. My next plan is to change it for a TeX plugin as it was announced. For that I need: * parsing of \showlists output, not very difficult * accessing tfm, it will take some time, I have to rewrite my NTS module * outputing TeX input So it will all take some time, but then it won't be dependent on ADvi and it will work under Windows too. So it makes more sense not to install ADvi and to wait few weeks. Cheers, --ksk

1 0

Karel's presentation on TUG 2005
by Karel Skoupý 29 Sep '05

29 Sep '05

Hi All, reacting to Thanh's request, I put my TUG 2005 presentation on http://www.inf.ethz.ch/personal/kaskoupy/slides/FreeShapeTextFormatting.pdf I doubt that there can be anything useful understood just from those slides. I should still write a proper paper in next few days, that should be much more clear and complete, you can safely wait for that. I'll also send you the actual prototype with the instructions how to get it running, hopefully tonight. Regards, --ksk

1 0

Re: [Dev-luatex] plugin for external formatting
by Thanh Han The 25 Sep '05

25 Sep '05

Hi, this is a long thread and too many things are discussed at the same time so I will need some time to read and understand what is going on. My first thought is that some small modifications to \showlist and \showbox will help a lot. It's easy to write additional info like dimensions of each item in the list, or in case of characters the filename of a tfm with fontsize (or we may write the dimensions of each char as Hans suggested, but this is an overkill IMHO). My feeling is that we need to work out the specification and format of the `` node list'' first. In the first step, I would prefer to have only node-specific things, eg only what comes out after a box construction. I also got a similar request: to provide a primitive that writes out the content of a box and another primitive to re-construct that box back from the output. We can start with this and make further extensions later on. At the moment I cannot see clearly what is needed, but I am willing to write some extensions so that we can experiment with to see what is really needed and perhaps change what have been done. Thanh On Tue, Sep 20, 2005 at 11:58:08AM +0200, Karel Skoupy wrote: > Hi all, > > as Hans has already mentioned, my concern with luatex is to have some > interface/protocol for formatting the TeX stuff externally. > > Without going to details now, I'm interested in alternative algorithms for > formatting not only paragraphs, but the whole stream. For TUG 2005 I have > written a prototype which doesn't use any TeX code at all (it just > parasities on ADvi code for getting some metric information and showing the > results). For long time I planned to make a whole new system from scratch, > but for several reasons, that was reconsidered and Hans proposed a way > (plugin mechanism for external engine), how to cooperate with TeX, so TeX > could benefit from the new algorithm and I can concentrate on the core > stuff. > > So basically I need a stream of (character) boxes, glues, penalties, ... > (is there a simple unambiguous notion for all that?) in a preprocessed > form (I don't care about input and macro handling) plus some parameters > (standard paragraph breaking parameters and the new special ones) and I > will return a stream of fixed boxes. > > 'I' will often mean 'the engine' depending on the context :-) > > In the first stage, I won't need lua (or any changes to TeX) at all. I > plan to use \showlists for my input stream and to generate a standard > TeX input file for reading the result back. Of course, it won't be so > simple, there will be some macro programming and trickery, which will > make the whole thing complicated, fragile, unreliable, and inefficient for > real use. Therefore some hooks from the actively developed TeX will be > probably useful for making the cooperation of TeX and the external > engine smooth. It might use lua or not, we will see, in any case I would > like to keep the plugin support generic and (complete but) minimal. > > I will now list the aspects of the communications between TeX and the > engine which I have thought of so far. I will be glad if you can just think > about it for the moment and give me some feedback if you will. > > * single paragraph stuff > > I need: > (1) complete representation of all the stuff which is to be returned formatted > (2) sizes of all the objects which are involved in formatting > (3) properties which influence the formatting (breakable, discardable, ...) > > It seems that the standard output of \showlists (or \showbox) will mostly do. > (1) is fulfilled I guess (the returned input needs to be only slightly > modified to fit TeX). > > (2) is little bit tricky, because for the characters I get only an id of > the font. So I will need to know the exact reference to a real font to get > the metrics information. This can be learned by eg. \show\tenrm. But of > course it is not know in advance what fonts are used in the paragraph, so > either all fonts can be listed at the beginning -- but where to get the list of > all font definitions, and the definitions can actually change in the middle > of the paragraph -- or I can make a first pass, collect the font ids and > ask for them in the second pass. It will be bit tricky and won't be > reliable due to redefinitions (I can also change the current id using \let > and lose the old id (still used in the log), right?), so it will be OK for > experimenting but for a real version, I will need a better support from > TeX. > > (3) is implicit, right? > > * stream of paragraphs > > I can need even the whole chapter, because I want to treat > - shapes and layouts, which are relative to page and not to a particular > paragraph > - pagination, floats placement > > For the basic experimenting I can redefine \par to something like > \hfil\break\indent but it will restrict all kinds of things which can > happen between the paragraphs (in vertical mode). Of course, the whole > thing will never be compatible to TeX, because TeX expect after \par that > the last paragraph was formatted and placed on the vertical list. So it > will be responsibility of the user/macro-programmer to bear the > consequences of using the alternative mechanism. Nevertheless, the > consequences should be as small as possible. > > So for the prototyping I can redefine \par or perhaps I can store the whole > paragraphs in infinite hboxes (redefining \hsize?) or maybe I can use some > \specials for tagging, but for the production version, this will be a very > tricky part. Not so much for the engine, but mainly on the TeX side. It > should be of a great concern for people who would want to use the new > algorithms in their systems (Hans?), (after those ideas are first tested by > a prototype :-). > > * passing the parameters specific to the new algorithms > > - layouts, shapes > - maybe others, like weights for resolving paragraph contra page breaking > > This will be a new thing so I hope that there is no compatibility burden. > > * hyphenation > > It will be a lot of additional work, but I think that I should handle it > locally. There are two reasons: > > (1) the protocol for failing and getting the list with new discretionaries > (TeX's 2nd pass) for every individual paragraph would be extremely > complicated, in the end it might be more difficult than handling it locally. > > (2) TeX's hyphenation mechanism is IMHO one of the crappiest parts of TeX. > I mean the way how the (non)ligatures are screwed up for discretionaries > which are not used in the end. So if it is handled locally, it will be IMO > simpler and more correct. There are also some research results concerning > hyphenation, which are not implemented in TeX, because it would be too > complicated. > > At the first stage, I'll omit the hyphenation completelly. > > At the moment, I don't remember anything else. I'm looking forward for your > feedback. > > --ksk > _______________________________________________ > Dev-luatex mailing list > Dev-luatex(a)ntg.nl > http://www.ntg.nl/mailman/listinfo/dev-luatex >

3 3

Re: [Dev-luatex] plugin for external formatting
by Thanh Han The 23 Sep '05

23 Sep '05

On Thu, Sep 22, 2005 at 09:02:43AM +0200, Hans Hagen wrote: > Thanh Han The wrote: > > >My feeling is that we need to work out the specification and > >format of the `` node list'' first. In the first step, I > >would prefer to have only node-specific things, eg only what > >comes out after a box construction. I also got a similar > >request: to provide a primitive that writes out the content > >of a box and another primitive to re-construct that box back > >from the output. We can start with this and make further > >extensions later on. > > > >At the moment I cannot see clearly what is needed, but I am > >willing to write some extensions so that we can experiment with > >to see what is really needed and perhaps change what have been done. > > > > > i'd say ... go ahead, so that we get a picture; at least we then hav ea > starting point for karl's work err -- we need to work out how the node list will be output and input back first. I am still waiting for Karel's opinion. Thanh

1 0

pdfliteral after text/rules
by Hartmut Henkel 22 Sep '05

22 Sep '05

Hi, here ist the problem, that the "page" mode of \pdfliteral and \latelua does not by itself guarantee, that the result goes relative to the lower left corner (the PDF origin): \nopagenumbers \pdfcompresslevel=0 X\latelua{ x={};y={}; c=72/72.27/65536; x[1]=pdf.geth()*c; y[1]=pdf.getv()*c } \input tufte X\latelua page{ x[2]=pdf.geth()*c; y[2]=pdf.getv()*c } \hrule \latelua page{ tex.print(x[1], " ", y[1], " m ", x[2], " ", y[2], " l S") } \input tufte \bye The above example works as intended only, if you remove the \hrule. Reason is, that for setting a rule, a transform "cm" is done away from the origin, then the rule is placed, but there is no transforms back to the origin. The next placements are done after a transform relative to the placement point of the rule... Text is always typeset with the transform matrix "cm" set to the origin, and inside the BT...ET group the Td operator does the local transform. There seems to be currently no \pdfliteral (or \latelua) mode that guarantees positioning relative to the PDF origin. But positioning relative to the origin would be nice to have since we now have the pdf.geth/v() lua functions. Else one would have to think what was the state (text or rule?) just before using \pdfliteral page... To solve this ambiguity, i have tried the attached patch, which puts any rule into a qQ group and does "cm" transforms inside this group. So the rules are handled like text, which is also grouped (inside BT...ET). That means, that the global working reference point is (hopefully :) always (?) at the origin. With the rule.ch patch the above example works apparently ok. This also means, that when using \pdfliteral page, one gets stuff placed at the origin, and still without transform (which should be ok). This would spare to invent yet another \pdfliteral mode. No idea how general it is. And wonder, how backward-compatible? We would need to check also \pdfrefximage... Just for discussion... Regards, Hartmut P.S. Let's see how the attachment is handled. Never tried this before.

3 5

plugin for external formatting
by Karel Skoupý 21 Sep '05

21 Sep '05

Hi all, as Hans has already mentioned, my concern with luatex is to have some interface/protocol for formatting the TeX stuff externally. Without going to details now, I'm interested in alternative algorithms for formatting not only paragraphs, but the whole stream. For TUG 2005 I have written a prototype which doesn't use any TeX code at all (it just parasities on ADvi code for getting some metric information and showing the results). For long time I planned to make a whole new system from scratch, but for several reasons, that was reconsidered and Hans proposed a way (plugin mechanism for external engine), how to cooperate with TeX, so TeX could benefit from the new algorithm and I can concentrate on the core stuff. So basically I need a stream of (character) boxes, glues, penalties, ... (is there a simple unambiguous notion for all that?) in a preprocessed form (I don't care about input and macro handling) plus some parameters (standard paragraph breaking parameters and the new special ones) and I will return a stream of fixed boxes. 'I' will often mean 'the engine' depending on the context :-) In the first stage, I won't need lua (or any changes to TeX) at all. I plan to use \showlists for my input stream and to generate a standard TeX input file for reading the result back. Of course, it won't be so simple, there will be some macro programming and trickery, which will make the whole thing complicated, fragile, unreliable, and inefficient for real use. Therefore some hooks from the actively developed TeX will be probably useful for making the cooperation of TeX and the external engine smooth. It might use lua or not, we will see, in any case I would like to keep the plugin support generic and (complete but) minimal. I will now list the aspects of the communications between TeX and the engine which I have thought of so far. I will be glad if you can just think about it for the moment and give me some feedback if you will. * single paragraph stuff I need: (1) complete representation of all the stuff which is to be returned formatted (2) sizes of all the objects which are involved in formatting (3) properties which influence the formatting (breakable, discardable, ...) It seems that the standard output of \showlists (or \showbox) will mostly do. (1) is fulfilled I guess (the returned input needs to be only slightly modified to fit TeX). (2) is little bit tricky, because for the characters I get only an id of the font. So I will need to know the exact reference to a real font to get the metrics information. This can be learned by eg. \show\tenrm. But of course it is not know in advance what fonts are used in the paragraph, so either all fonts can be listed at the beginning -- but where to get the list of all font definitions, and the definitions can actually change in the middle of the paragraph -- or I can make a first pass, collect the font ids and ask for them in the second pass. It will be bit tricky and won't be reliable due to redefinitions (I can also change the current id using \let and lose the old id (still used in the log), right?), so it will be OK for experimenting but for a real version, I will need a better support from TeX. (3) is implicit, right? * stream of paragraphs I can need even the whole chapter, because I want to treat - shapes and layouts, which are relative to page and not to a particular paragraph - pagination, floats placement For the basic experimenting I can redefine \par to something like \hfil\break\indent but it will restrict all kinds of things which can happen between the paragraphs (in vertical mode). Of course, the whole thing will never be compatible to TeX, because TeX expect after \par that the last paragraph was formatted and placed on the vertical list. So it will be responsibility of the user/macro-programmer to bear the consequences of using the alternative mechanism. Nevertheless, the consequences should be as small as possible. So for the prototyping I can redefine \par or perhaps I can store the whole paragraphs in infinite hboxes (redefining \hsize?) or maybe I can use some \specials for tagging, but for the production version, this will be a very tricky part. Not so much for the engine, but mainly on the TeX side. It should be of a great concern for people who would want to use the new algorithms in their systems (Hans?), (after those ideas are first tested by a prototype :-). * passing the parameters specific to the new algorithms - layouts, shapes - maybe others, like weights for resolving paragraph contra page breaking This will be a new thing so I hope that there is no compatibility burden. * hyphenation It will be a lot of additional work, but I think that I should handle it locally. There are two reasons: (1) the protocol for failing and getting the list with new discretionaries (TeX's 2nd pass) for every individual paragraph would be extremely complicated, in the end it might be more difficult than handling it locally. (2) TeX's hyphenation mechanism is IMHO one of the crappiest parts of TeX. I mean the way how the (non)ligatures are screwed up for discretionaries which are not used in the end. So if it is handled locally, it will be IMO simpler and more correct. There are also some research results concerning hyphenation, which are not implemented in TeX, because it would be too complicated. At the first stage, I'll omit the hyphenation completelly. At the moment, I don't remember anything else. I'm looking forward for your feedback. --ksk

3 9

lua keyword { }
by Hans Hagen 18 Sep '05

18 Sep '05

Hi, So far we have \lua {...} : expanded, result of tex.print is inserted with catcodes 11/12 \lua otr {...} : delayed expansion (in otr), only pdf.print is used, tex.print is ignored currently we need to use \scantokens to process the result of tex.print in a texie way (current catcode regime). So, in addition we can have: \lua tex {...} : expanded, result of tex.print is fed into internal scantokens This saves passing args around. we also need: \luatexversion (currently 0.1.5) \luaversion (currently 5.0.2) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

2 6

FWD: Subversion Repository
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: lua Date: Tue, 13 Sep 2005 15:37:42 +0200 From: Taco Hoekwater <taco(a)elvenkind.com> To: Taco Hoekwater <taco(a)elvenkind.com> CC: Hans Hagen <pragma(a)wxs.nl>, Hartmut Henkel <hartmut_henkel(a)gmx.de> References: <4325FF95.2080808(a)wxs.nl> <Pine.LNX.4.61.0509130030060.5488(a)hahepc1.hahe> <4326075F.9030301(a)wxs.nl> <43267768.1020409(a)elvenkind.com> <43268242.5010306(a)wxs.nl> <4326878E.6070300(a)elvenkind.com> <432692BB.7070806(a)elvenkind.com> Hi, Ok, I've committed what is essentially * pdftex 1.30.3 (from sarovar) * the src directory from the lua5.1-alpha release (lua.org) * lua.ch / luastuff.c (our sources) The build system is just like pdftex, and it 'works for me'. Also, I've added Hans to the project. Main project page: https://foundry.supelec.fr/projects/luatex/ Cheers, Taco Taco Hoekwater wrote: > Hi again, > > The project is approved already :-) > > If you both register as users, then I can add you to the project. > > https://foundry.supelec.fr/ > > > Taco Hoekwater wrote: > >> >> I've requested a project at supelec. I assume Fabrice will approve >> that in a day or so, and then I will upload sources there (based >> on the "standalone" pdfetex from sarovar). > >

1 0

FWD: Re: The Other LuateX
by Taco Hoekwater 15 Sep '05

15 Sep '05

-------- Original Message -------- Subject: Re: LuateX Date: Thu, 15 Sep 2005 08:55:11 +0200 From: Taco Hoekwater <taco(a)elvenkind.com> To: Hans Hagen <pragma(a)wxs.nl> CC: Hartmut Henkel <hartmut_henkel(a)gmx.de> References: <20050912140052.GE17097(a)lucien.oneiros.kn-bremen.de> <432592A7.3000708(a)wxs.nl> <20050912154605.GG17097(a)lucien.oneiros.kn-bremen.de> <4325A568.3060300(a)wxs.nl> <Pine.LNX.4.61.0509122047420.4490(a)hahepc1.hahe> <4325EA6B.9050206(a)wxs.nl> <Pine.LNX.4.61.0509122330070.5143(a)hahepc1.hahe> <Pine.LNX.4.61.0509130002150.5327(a)hahepc1.hahe> <4325FF12.1070906(a)wxs.nl> <Pine.LNX.4.61.0509140000590.16751(a)hahepc1.hahe> <432756B3.10307(a)wxs.nl> <Pine.LNX.4.61.0509142201430.4684(a)hahepc1.hahe> <432886F0.9020304(a)wxs.nl> Hans Hagen wrote: > Hartmut Henkel wrote: > >> just googled for luatex. To my surpries, many hits... >> >> http://www.luatex.com/ Unfortunate, I should have checked for that before registering the new domain. :-( >> > ha, interesting, > > so we nee to change to: > pdfluatex I much favor this solution, because it allows the continued use of the domain as well as the foundry project, and most of the other proposals were a bit too short for my taste :-) Taco

1 0