Jonathan Fine wrote:

As subject. xdv is the eXtended DVi format used by XeTeX.

No, it cannot. Perhaps it should, but that needs a volunteer. Best wishes, Taco

Arthur Reutenauer wrote:

...

As subject. xdv is the eXtended DVi format used by XeTeX.

Actually, it can. LuaTeX produces DVI, which is perfectly palatable to XeTeX's xdv2pdf and xdvipdfmx. What features of xdv are you missing in LuaTeX's output?

Knuth and MacKay were the first to extent DVI, because it does not adequately support bidirectional typesetting. Are you saying, Arthur, that Source to PDF using LuaTeX directly can proceed via Source to DVI using LuaTeX and then DVI to PDF using xdv2pdf (or some other tool) ? I've been told (by Hans Hagen) that even for pdfTeX this is not advisable. I'm looking for a what might be called a Unicode savvy Device Independent binary format. And I'm looking for XeTeX and LuTeX to share code and ideas, when possible. I'm copying Jonathan Kew (XeTeX's developer in on this). -- Jonathan

Knuth and MacKay were the first to extent DVI, because it does not adequately support bidirectional typesetting.

Yes, and, as I already tried to communicate to you, this extended format (I assume you mean DVI-IVD) seems to me like a dead end, because at the time it was developed, over twenty years ago, there was almost no foundation for encoding bidirectional text; and it has never been updated to take Unicode's bidi algorithm into account.

Are you saying, Arthur, that Source to PDF using LuaTeX directly can proceed via Source to DVI using LuaTeX and then DVI to PDF using xdv2pdf (or some other tool) ?

Of course it can, depending on the information you put in your DVI. If you've been attentive to the explanations I sent earlier to this list about vertical typesetting, you'll know where it can fail. In this specific case, though, xdv's extensions would have been of no use at all: we would need some further extensions of the DVI format.

I've been told (by Hans Hagen) that even for pdfTeX this is not advisable.

It certainly isn't advisable. Nevertheless, in many cases, it's still possible.

I'm looking for a what might be called a Unicode savvy Device Independent binary format. And I'm looking for XeTeX and LuTeX to share code and ideas, when possible.

Hence, what you're aiming at is for LuaTeX and XeTeX to produce some common extended DVI format, not for LuaTeX to produce xdv, right? This is quite different since, as I said above, LuaTeX has some features that XeTeX's xdv can't account for.

I'm copying Jonathan Kew (XeTeX's developer in on this).

I know who Jonathan Kew is. Arthur

Arthur Reutenauer wrote: [snip]

...

I'm looking for a what might be called a Unicode savvy Device Independent binary format. And I'm looking for XeTeX and LuTeX to share code and ideas, when possible.

Hence, what you're aiming at is for LuaTeX and XeTeX to produce some common extended DVI format, not for LuaTeX to produce xdv, right? This is quite different since, as I said above, LuaTeX has some features that XeTeX's xdv can't account for.

I think you've got the gist of what I want. Sharing of code and ideas. However, your statement seems to contradict your earlier statement (snipped) that one can do Source transformed by LuaTeX to DVI and hence PDF and get the same result as Source transformed directly to PDF by LuaTeX From what you've said, it seems to me that xdv contains some but not all of the features that LuaTeX would need to add to the DVI format. Are there many such features?

...

I'm copying Jonathan Kew (XeTeX's developer in on this).

I know who Jonathan Kew is.

Good. I'd be surprised and worried if you didn't. In fact, I know that you and he have probably met and, I hope, discussed some of these matters, because the three of us were all at TUG 2008 in Cork last year (yes, it is 2009 now). But I put this statement in my message not for you but for others on this list who might not recall that Jonathan Kew is XeTeX's developer. -- Jonathan

On Thu, Jan 1, 2009 at 11:09 PM, Jonathan Fine wrote:

What I want is for LuaTeX and XeTeX to have a shared 'extended dvi format' which is suitable for print, for generating PDF and possibly other purposes.

Just curious: how do you print xdv (what's wrong with printing pdf?) and what other purposes do you (or would you) use xdv for? Mojca

David Kastrup writes:

So I don't think that the DVI format does not make for a sensible starting point for embedding such information.

Read what I mean, not what I write. -- David Kastrup, Kriemhildstr. 15, 44793 Bochum

Jonathan Fine wrote:

What I want is for LuaTeX and XeTeX to have a shared 'extended dvi format' which is suitable for print, for generating PDF and possibly other purposes.

And of course I'd like this format to be of at least satisfactory technical quality.

Since luatex is mostly pdftex the code related to the backend is also interwoven with functionality (like annotations and objects etc). Most if that can be supported via the dvi route too but it's not always as efficient. Also, some functionality can be written in lua by parsing node lists etc. Anyhow, it's on the agenda (2009-2010) to rewrite the backend code in such a way that more backend can be used (maybe written in lua). Although we want luatex to be backward compatible as much as possible (as long as it does not complicated the code), it might be that redesigning the interface between frontend and backend will lead to extensions to the dvi format (or we might as well decide that dvi supports only part of the frontend features); specials are always an escape at the cost of more macro package code. So, it's not 'dvi' that sits in the middle of 'print and generating pdf', but dvi as one of the (traditional) output formats: luatex frontend -> api -> [dvi,pdf,...] [dvi->ps,pdf,...] of course one can stil go the luatex frontend -> api -> [dvi+] [ps,pdf,...] route then. As said: it's one of the longer term objectives. A quick and dirty non-introsve patch to produce xetex compatible dvi is no problem, but one never knows if it will stay that way. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Martin Schröder wrote:

2009/1/1 Jonathan Fine :

...

What I want is for LuaTeX and XeTeX to have a shared 'extended dvi format' which is suitable for print, for generating PDF and possibly other purposes.

Thus loosing the benefits of directly producing PDF. I don't see many benefits here.

Please could I be told, what are the main benefits of directly producing PDF? BTW, Jonathan Kew said recently (on c.t.t) that XeTeX can use two processors on a multicore machine precisely because typesetting and PDF generation are decoupled, via xdv. -- Jonathan

Taco Hoekwater wrote: [discussion of plans snipped] PDF is very strong in the print field. However, the web page is also an important medium, and as I said in my response to Mojca, I'd like to be able to view and interact with typeset material on a more-or-less ordinary web page. I'd also like to have TeX or an extension available as a typesetting engine for an interactive graphical typesetting program. For these reasons, and also so that LuaTeX and XeTeX could share more code and ideas, I'd like both systems to use a common extension of DVI. In some sense, it is this sharing of code and of ideas that is for me the key thing.

PS And a Happy New Year to you all!

Well said. And from me to. -- Jonathan

Talking about DVI and PDF, I am the first one to be concerned about the programmed extinction of DVI because I have many jobs based on DVI post-processing (for example for marginal material, headers, and even for parallel texts between two pages). The advantage with DVI is that you can very easily now where you are located on the page so that you can extract text with positional information, and you can also easily insert new text (using pushes and pops). But I'm confident that I will be able to rewrite all that for the PDF format. Multivalent has released some very powerful Java classes which convert PDF into a human-readable format and back to binary. Of course BT...ET sections are more complex than DVI's SET operator but I think it is possible to do simple post-processing. And with the Mars format there is always hope that all that will be written in XML... So, although many of the tools I depend upon are based on DVI, I'm rather for a PDF-only solution. Nevertheless, a problem that should be solved is the one of adapting psfrag to PDF. psfrag is *absolutely essential* when typesetting mathematics... Le 2 janv. 09 à 19:52, Jonathan Fine a écrit :

Taco Hoekwater wrote:

[discussion of plans snipped]

PDF is very strong in the print field. However, the web page is also an important medium, and as I said in my response to Mojca, I'd like to be able to view and interact with typeset material on a more- or-less ordinary web page.

I'd also like to have TeX or an extension available as a typesetting engine for an interactive graphical typesetting program.

For these reasons, and also so that LuaTeX and XeTeX could share more code and ideas, I'd like both systems to use a common extension of DVI.

In some sense, it is this sharing of code and of ideas that is for me the key thing.

...

PS And a Happy New Year to you all!

Well said. And from me to.

-- Jonathan

_______________________________________________ dev-luatex mailing list dev-luatex@ntg.nl http://www.ntg.nl/mailman/listinfo/dev-luatex

-- + -----------------------------------------------------------------------+ | Yannis Haralambous, Ph.D. yannis.haralambous@telecom- bretagne.eu | | Directeur d'Études http://omega.enstb.org/ yannis | | Tel. +33 (0)2.29.00.14.27 | | Fax +33 (0)2.29.00.12.82 | | Département Informatique | | TELECOM Bretagne | | Technopôle de Brest Iroise, CS 83818, 29238 Brest CEDEX 3, France | | Coordonnées Google-Earth : 48°21'31.57"N 4°34'16.76"W | + -----------------------------------------------------------------------+ ...pour distinguer l'extérieur d'un aquarium, mieux vaut n'être pas poisson ...the ball I threw while playing in the park has not yet reached the ground Es gab eine Zeit, wo ich nur ungern über Schubert sprechen, nur Nächtens den Bäumen und Sternen von ihm vorerzählen mögen.

Yannis Haralambous wrote:

Talking about DVI and PDF, I am the first one to be concerned about the programmed extinction of DVI because I have many jobs based on DVI post-processing (for example for marginal material, headers, and even for parallel texts between two pages). The advantage with DVI is that you can very easily now where you are located on the page so that you can extract text with positional information, and you can also easily insert new text (using pushes and pops).

i'm not sure if you refer to the same kind of positional info, but pdftex (and therefore luatex) hasn pdfsavepos cum suis in both pdf and dvi mode so you can store positions (lazy, i.e. write them to file in the backend) and then use them in a second pass (it't a rather old feature by now, which started as an experiment by Thanh and me and for some time was only available in pdf mode and in dvi mode we could use specials and the dvipos postprocessor; after a while it was also added to dvi mode) on the agenda is some more positional (runtime, in prerolls using boxes and getting pos into afterwards), a kind of pet project in parallel

And with the Mars format there is always hope that all that will be written in XML...

and xps .. although one might have problems then with packages resources (and files become large too) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Le 2 janv. 09 à 21:49, Hans Hagen a écrit :

i'm not sure if you refer to the same kind of positional info, but pdftex (and therefore luatex) hasn pdfsavepos cum suis in both pdf and dvi mode so you can store positions (lazy, i.e. write them to file in the backend) and then use them in a second pass (it't a rather old feature by now, which started as an experiment by Thanh and me and for some time was only available in pdf mode and in dvi mode we could use specials and the dvipos postprocessor; after a while it was also added to dvi mode)

on the agenda is some more positional (runtime, in prerolls using boxes and getting pos into afterwards), a kind of pet project in parallel

Well, I'm doing mainly two things: 1) I put tags in the DVI file which allow me to place marginal material at (exactly) the same height, during post-processing. In DVI it is easy to place a PUSH, make a skip, typeset the material and then POP back to the previous position so that the rest of the page remains unchanged. I guess this will be harder in PDF. 2) I calculate the height of lines so that I can detect whether a page is shorter or taller than the normal. This is very useful to prevent accidents. In a more elaborate project I split my DVI page into lines (lines of text, of translation, of footnotes of translation and of one or more critical apparati) and I re-assemble these lines into double pages, by letting the critical apparati flow under the footnotes, ad by keeping parallelism between text and translation. This works fine with DVI, maybe everyline can become a BT...ET section in PDF and I can get the same result. -- + -----------------------------------------------------------------------+ | Yannis Haralambous, Ph.D. yannis.haralambous@telecom- bretagne.eu | | Directeur d'Études http://omega.enstb.org/ yannis | | Tel. +33 (0)2.29.00.14.27 | | Fax +33 (0)2.29.00.12.82 | | Département Informatique | | TELECOM Bretagne | | Technopôle de Brest Iroise, CS 83818, 29238 Brest CEDEX 3, France | | Coordonnées Google-Earth : 48°21'31.57"N 4°34'16.76"W | + -----------------------------------------------------------------------+ ...pour distinguer l'extérieur d'un aquarium, mieux vaut n'être pas poisson ...the ball I threw while playing in the park has not yet reached the ground Es gab eine Zeit, wo ich nur ungern über Schubert sprechen, nur Nächtens den Bäumen und Sternen von ihm vorerzählen mögen.

However, your statement seems to contradict your earlier statement (snipped) that one can do Source transformed by LuaTeX to DVI and hence PDF and get the same result as Source transformed directly to PDF by LuaTeX

I didn't write that you would get the same result. But in most cases, it should still work, though. Again, it all depends on what kind of information you want to put it your output file, so you should be more specific.

From what you've said, it seems to me that xdv contains some but not all of the features that LuaTeX would need to add to the DVI format. Are there many such features?

I couldn't say for sure, but if you go back to my vertical text sample, you can see an essential feature that DVI is lacking, as well as xdv: the ability to include arbitrarily modified metrics for any font. In my example, I want to modify the dimensions of glyphs in a font, much like you would do with a virtual font. LuaTeX allows us to emulate this behaviour by using the define_font callback, without going through an actual .vf file. You need, roughly, to produce a Lua hash table in which you put new widths, heights and depths for all the glyphs in that font. This only works to a certain extent in DVI mode, because, then, I would like to modify the vertical placement of some glyphs, and you can't include that information in the DVI. Inside LuaTeX, all that information is contained in your Lua code, but it is lost once you write the DVI file. Needless to say, it would also be lost in the xdv format. In short, this demontrates that DVI and extensions lack an essential feature of LuaTeX, which is, not very surprisingly, Lua itself. Should we go on adding the possibility of including arbitrary Lua code in a DVI file? I shouldn't think so. The way I see it, Lua is part of the typesetting engine of LuaTeX, not the end format (whichever one you use), and whatever you do during the typesetting part should be discarded in the backend, and transformed into the appropriate form (in my example, by writing out the modified metrics in the PDF file, and shifting the glyphs in the embedded font). Incidentally, you can probably still try and include arbitrary Lua code in a DVI file, by using \special's. You can then produce a DVI consumer application that would read and execute that code (for example, by modifying xdvipdfmx). But if you go on producing PDF instead, everything is already there for you. Arthur

Martin Schröder wrote:

2009/1/2 Jonathan Fine :

...

Please could I be told, what are the main benefits of directly producing PDF?

Reasoning about and manipulating included PDFs. This is much harder with a two-pass engine like XeTeX + postprocessor; as an example compare the knowledge of XeTeX (the program) and pdfTeX about included PDFs.

I'm sorry, but I don't understand. Yes, typesetting applications need to know about included objects such as images (and fonts). But from there you conclude that there's a benefit in generating PDF directly. I don't see how this follows. I'll invent an example (so it's not really an example). We wish to resize an image to fit the measure, but if it's a bitmap image we want to choose a scaling factor that works well with our planned output device. Here's another. We want to know if the image contains colour, because if it does it should appear in the colour plates section. In both cases we have to open the image file and, as you say, reason about it. Based on this reasoning we come to a conclusion (such as a scale factor) and both 9a) typeset the text and (b) process the image accordingly. LuaTeX does (a) and (b) at the same time. I see no reason why a 'special' that says 'do (b)' should not be written to the xdv or whatever output file. [Discussion of performance snipped]

But as I said: Show us your code.

Well, it's not MY code, but I'm sure you're welcome to use it: http://scripts.sil.org/svn-public/xetex/TRUNK/ http://scripts.sil.org/svn-public/xdvipdfmx/TRUNK/ XeTeX is distributed under the X11 free software license. -- Jonathan

Jonathan Fine skribis:

Well, it's not MY code, but I'm sure you're welcome to use it: http://scripts.sil.org/svn-public/xetex/TRUNK/ http://scripts.sil.org/svn-public/xdvipdfmx/TRUNK/

XeTeX is distributed under the X11 free software license.

xdvipdfmx is GPL, I believe.

Yes, typesetting applications need to know about included objects such as images (and fonts). But from there you conclude that there's a benefit in generating PDF directly. I don't see how this follows.

The key to understanding the “benefit in generating PDF directly” lies in the adverb “directly”, not the noun “PDF”. We appreciate having a single program that has all the necessary knowledge about the end result.

LuaTeX does (a) and (b) at the same time.

Not at the same time, within the same program. Those actions do not need to be carried out at the same moment, but it's useful that the parts of the program that are responsible for them are able to communicate with each other to exchange knowledge about the material being typeset, without duplicating code, as Martin said.

I see no reason why a 'special' that says 'do (b)' should not be written to the xdv or whatever output file.

Maybe it's time that we turn the question the other way round and ask you for reasons to *do* so. What, could you please tell us, is so great about *not* generating the end result directly that we should force ourselves to produce an intermediate file first, and even go on about inventing a new format, because the current extensions to DVI lack some necessary features?

[Discussion of performance snipped]

It's a pity you don't want to discuss this, because it's the only technical advantage to producing xdv that I have seen mentioned so far.

Well, it's not MY code, but I'm sure you're welcome to use it: http://scripts.sil.org/svn-public/xetex/TRUNK/ http://scripts.sil.org/svn-public/xdvipdfmx/TRUNK/

That's XeTeX's code. The LuaTeX developers are well aware of XeTeX's existence and features, but what you may be missing here is that LuaTeX is not XeTeX and is not destined to be, nor the other way round, for that matter. They simply serve different purposes. Your insistence in having LuaTeX produce xdv seems to proceed from the assumption that XeTeX's novelty resides in xdv, which is in my opinion the wrong angle on this issue: XeTeX's most valuable asset is XeTeX, the program, itself, not xdv production. And xdvipdfmx, the postprocessor, doesn't have such prominent features that couldn't be implemented within LuaTeX, as Barry pointed out. As for LuaTeX's specific features, they would not be very well expressed by xdv, and can be fully accounted for by generating PDF. Arthur