the original DVI format already supports 4-byte character, what more Unicode-savviness do you need? Of course one should decide whether DVI should contain glyph indexes or Unicode codepoints. Unless you mean by Unicode-savviness that one should have both glyph indexes and Unicode codepoints (I think that's what XeTeX is doing). A very bad approach since character and glyph correspondances are many- to-many. On the other hand, if we use just glyph indexes in the DVI file then we can hardly do anything with it than print it (going to PDF will be difficult since the correspondance with characters is lost). Le 1 janv. 09 à 21:43, Jonathan Fine a écrit :

I'm looking for a what might be called a Unicode savvy Device Independent binary format. And I'm looking for XeTeX and LuTeX to share code and ideas, when possible.

-- + -----------------------------------------------------------------------+ | Yannis Haralambous, Ph.D. yannis.haralambous@telecom- bretagne.eu | | Directeur d'Études http://omega.enstb.org/ yannis | | Tel. +33 (0)2.29.00.14.27 | | Fax +33 (0)2.29.00.12.82 | | Département Informatique | | TELECOM Bretagne | | Technopôle de Brest Iroise, CS 83818, 29238 Brest CEDEX 3, France | | Coordonnées Google-Earth : 48°21'31.57"N 4°34'16.76"W | + -----------------------------------------------------------------------+ ...pour distinguer l'extérieur d'un aquarium, mieux vaut n'être pas poisson ...the ball I threw while playing in the park has not yet reached the ground Es gab eine Zeit, wo ich nur ungern über Schubert sprechen, nur Nächtens den Bäumen und Sternen von ihm vorerzählen mögen.

On Thu, Jan 1, 2009 at 11:09 PM, Jonathan Fine wrote:

What I want is for LuaTeX and XeTeX to have a shared 'extended dvi format' which is suitable for print, for generating PDF and possibly other purposes.

Just curious: how do you print xdv (what's wrong with printing pdf?) and what other purposes do you (or would you) use xdv for? Mojca

Jonathan Fine writes:

Yannis Haralambous wrote:

...

the original DVI format already supports 4-byte character, what more Unicode-savviness do you need? Of course one should decide whether DVI should contain glyph indexes or Unicode codepoints.

I'd like bidirectional information to be available.

Fundamentally DVI describes isolated graphemes on paper. No sequence is implied, no relation between glyphs, most certainly not directional. Not even spaces are discernible, nor is it possible to rejoin hyphenated words and/or line breaks or even do cut&paste except by heuristics. So I don't think that the DVI format does not make for a sensible starting point for embedding such information. -- David Kastrup, Kriemhildstr. 15, 44793 Bochum

2009/1/1 Jonathan Fine :

What I want is for LuaTeX and XeTeX to have a shared 'extended dvi format' which is suitable for print, for generating PDF and possibly other purposes.

Thus loosing the benefits of directly producing PDF. I don't see many benefits here. But of course patches implementing this will be considered. Best Martin

Jonathan Fine wrote:

Please could I be told, what are the main benefits of directly producing PDF?

Convenience since all is packages in one file; also, it's in the spirit of good old tex to adapt to developments (like pdf). Nowhere is demanded that dvi is to be the output (in a similar fashion: specials and writes are examplex of extensions, and pdftex (luatex) adds a couple more. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Martin Schröder wrote:

2009/1/2 Jonathan Fine :

...

Please could I be told, what are the main benefits of directly producing PDF?

Reasoning about and manipulating included PDFs. This is much harder with a two-pass engine like XeTeX + postprocessor; as an example compare the knowledge of XeTeX (the program) and pdfTeX about included PDFs.

I'm sorry, but I don't understand. Yes, typesetting applications need to know about included objects such as images (and fonts). But from there you conclude that there's a benefit in generating PDF directly. I don't see how this follows. I'll invent an example (so it's not really an example). We wish to resize an image to fit the measure, but if it's a bitmap image we want to choose a scaling factor that works well with our planned output device. Here's another. We want to know if the image contains colour, because if it does it should appear in the colour plates section. In both cases we have to open the image file and, as you say, reason about it. Based on this reasoning we come to a conclusion (such as a scale factor) and both 9a) typeset the text and (b) process the image accordingly. LuaTeX does (a) and (b) at the same time. I see no reason why a 'special' that says 'do (b)' should not be written to the xdv or whatever output file. [Discussion of performance snipped]

But as I said: Show us your code.

Well, it's not MY code, but I'm sure you're welcome to use it: http://scripts.sil.org/svn-public/xetex/TRUNK/ http://scripts.sil.org/svn-public/xdvipdfmx/TRUNK/ XeTeX is distributed under the X11 free software license. -- Jonathan

2009/1/4 Jonathan Fine :

Martin Schröder wrote:

...

2009/1/2 Jonathan Fine :

...

Please could I be told, what are the main benefits of directly producing PDF?

Reasoning about and manipulating included PDFs. This is much harder with a two-pass engine like XeTeX + postprocessor; as an example compare the knowledge of XeTeX (the program) and pdfTeX about included PDFs.

I'm sorry, but I don't understand.

I think you do. :-)

I'll invent an example (so it's not really an example). We wish to resize an image to fit the measure, but if it's a bitmap image we want to choose a scaling factor that works well with our planned output device. Here's another. We want to know if the image contains colour, because if it does it should appear in the colour plates section.

In both cases we have to open the image file and, as you say, reason about it. Based on this reasoning we come to a conclusion (such as a scale factor) and both 9a) typeset the text and (b) process the image accordingly.

LuaTeX does (a) and (b) at the same time. I see no reason why a 'special' that says 'do (b)' should not be written to the xdv or whatever output file.

A special makes that harder; it needs more code. The backend needs the code for reasoning about the image anyway; now the frontend also needs it. You're duplicating code. And you need code for the two parts to communicate. More code means more errors. Consider this example: I want to include PDFs with layers and want to mix layers generated by the document with layers from the PDFs, controlled by the document. This might be doable in two passes, but it is much easier in one pass.

[Discussion of performance snipped]

...

But as I said: Show us your code.

Well, it's not MY code, but I'm sure you're welcome to use it:

You don't understand, so I'll be more clear: Shut up and code! :-) You seem to be the only one here interested in this mythical luatex + xdv, so why don't you SHOW us what it can do better? I can see only ONE benefit from your mythical xdv: interchangable backends. But I don't think the team finds *dvipdfm* so much more interesting than luaTeX. Best Martin

Arthur Reutenauer writes:

Maybe it's time that we turn the question the other way round and ask you for reasons to *do* so.

Hi Arthur, as far as I remember, Jonathan explained clearly at the very beginning of the discussion why he is interested in an intermediate format.

What, could you please tell us, is so great about *not* generating the end result directly that we should force ourselves to produce an intermediate file first, [...]

What do you mean with "end result"? Unless I completely misunderstand everything, Jonathan does *not* intend to create PDF at all. I assume that what Jonathan has in mind is something like http://www.mathtran.org/toys/jfine/editor2.html Yes, the PDF backend is extremely expensive in this case, for several reasons. Regards, Reinhard -- ---------------------------------------------------------------------------- Reinhard Kotucha Phone: +49-511-3373112 Marschnerstr. 25 D-30167 Hannover mailto:reinhard.kotucha@web.de ---------------------------------------------------------------------------- Microsoft isn't the answer. Microsoft is the question, and the answer is NO. ----------------------------------------------------------------------------

Knuth and MacKay were the first to extent DVI, because it does not adequately support bidirectional typesetting.

Yes, and, as I already tried to communicate to you, this extended format (I assume you mean DVI-IVD) seems to me like a dead end, because at the time it was developed, over twenty years ago, there was almost no foundation for encoding bidirectional text; and it has never been updated to take Unicode's bidi algorithm into account.

Are you saying, Arthur, that Source to PDF using LuaTeX directly can proceed via Source to DVI using LuaTeX and then DVI to PDF using xdv2pdf (or some other tool) ?

Of course it can, depending on the information you put in your DVI. If you've been attentive to the explanations I sent earlier to this list about vertical typesetting, you'll know where it can fail. In this specific case, though, xdv's extensions would have been of no use at all: we would need some further extensions of the DVI format.

I've been told (by Hans Hagen) that even for pdfTeX this is not advisable.

It certainly isn't advisable. Nevertheless, in many cases, it's still possible.

I'm looking for a what might be called a Unicode savvy Device Independent binary format. And I'm looking for XeTeX and LuTeX to share code and ideas, when possible.

Hence, what you're aiming at is for LuaTeX and XeTeX to produce some common extended DVI format, not for LuaTeX to produce xdv, right? This is quite different since, as I said above, LuaTeX has some features that XeTeX's xdv can't account for.

I'm copying Jonathan Kew (XeTeX's developer in on this).

I know who Jonathan Kew is. Arthur

However, your statement seems to contradict your earlier statement (snipped) that one can do Source transformed by LuaTeX to DVI and hence PDF and get the same result as Source transformed directly to PDF by LuaTeX

I didn't write that you would get the same result. But in most cases, it should still work, though. Again, it all depends on what kind of information you want to put it your output file, so you should be more specific.

From what you've said, it seems to me that xdv contains some but not all of the features that LuaTeX would need to add to the DVI format. Are there many such features?

I couldn't say for sure, but if you go back to my vertical text sample, you can see an essential feature that DVI is lacking, as well as xdv: the ability to include arbitrarily modified metrics for any font. In my example, I want to modify the dimensions of glyphs in a font, much like you would do with a virtual font. LuaTeX allows us to emulate this behaviour by using the define_font callback, without going through an actual .vf file. You need, roughly, to produce a Lua hash table in which you put new widths, heights and depths for all the glyphs in that font. This only works to a certain extent in DVI mode, because, then, I would like to modify the vertical placement of some glyphs, and you can't include that information in the DVI. Inside LuaTeX, all that information is contained in your Lua code, but it is lost once you write the DVI file. Needless to say, it would also be lost in the xdv format. In short, this demontrates that DVI and extensions lack an essential feature of LuaTeX, which is, not very surprisingly, Lua itself. Should we go on adding the possibility of including arbitrary Lua code in a DVI file? I shouldn't think so. The way I see it, Lua is part of the typesetting engine of LuaTeX, not the end format (whichever one you use), and whatever you do during the typesetting part should be discarded in the backend, and transformed into the appropriate form (in my example, by writing out the modified metrics in the PDF file, and shifting the glyphs in the embedded font). Incidentally, you can probably still try and include arbitrary Lua code in a DVI file, by using \special's. You can then produce a DVI consumer application that would read and execute that code (for example, by modifying xdvipdfmx). But if you go on producing PDF instead, everything is already there for you. Arthur

Jonathan Fine skribis:

But DVI lacks an essential feature of TeX, which is the TeX macro language. I count this as a benefit of DVI, not a deficiency. Standards restrict, and by restricting enable.

This may be a case of 'If you build it, they will come': present a set of tools making XDVI useful, and then maybe people will want to write code for it. Having written the clumsy XDVI support for ant (ant.berlios.de) a few years ago, and somewhat less clumsy support for unpublished software, I am trying to recall in what important ways XDVI differs from regular DVI. I _think_ (my memory may be wrong) the deal was that to use a non-tfm font you had to use the new glyph-writing opcodes. Otherwise it's almost exactly like using dvipdfm. I don't know what XeTeX does but in both the cases mentioned above I use glyph-index encoding. To make that portable you would have to include in the xdv file a map from glyph indexes to Unicode strings, perhaps using specials.

Hi all, Arthur Reutenauer wrote:

...

As subject. xdv is the eXtended DVi format used by XeTeX.

Actually, it can. LuaTeX produces DVI, which is perfectly palatable to XeTeX's xdv2pdf and xdvipdfmx. What features of xdv are you missing in LuaTeX's output?

As Barry said in another message, ToUnicode information would be useful. It is true that every DVI is also XDV, but not every XDV is DVI, and (some of) XeTeX's extensions would be nice to have if they can be integrated without too much effort. When it comes to new extensions, there already is at least one "Unicode-aware extended DVI format" commonly available, and that is ISO 32000, a.k.a. PDF 1.7. Note that from the user's point of view, a PDF is also what XeTeX generates. The backend conversion is hidden inside the executable. In the luatex engine, that already knows how to generate PDF on its own, there is no reason to go via DVI and (x)dvipdfmx to generate a PDF document. Also, the new PDF previewers like the one in texworks are fast enough to replace DVI previewers in interactive editing. Metapost is probably the last program that actually needs DVI for the label typesetting, and that dependency is already planned for removal. So the usefulness of DVI is limited and diminishing. In my opinion, it would therefore be a waste of effort to spend a lot of time working on an extension to the DVI backend. In fact, I advocate removing the whole possibility to create DVI output at some point in the future. Best wishes, Taco PS And a Happy New Year to you all!

Jonathan Fine wrote:

Taco Hoekwater wrote:

[discussion of plans snipped]

PDF is very strong in the print field. However, the web page is also an important medium, and as I said in my response to Mojca, I'd like to be

pdf is also quite strong in the preview on the web field; it packages all the resources in the file which is quite different from dvi

able to view and interact with typeset material on a more-or-less ordinary web page.

well, go ahead then and extend your favourite tex variant then; as far as i understood you already did something like that; also keep in mind that luatex, becoming a flexible machine, most likely becomes slower than faster; maybe next year (when we're further with redoing the backend) we might discuss dvi again;

I'd also like to have TeX or an extension available as a typesetting engine for an interactive graphical typesetting program.

For these reasons, and also so that LuaTeX and XeTeX could share more code and ideas, I'd like both systems to use a common extension of DVI.

well, it makes no sense to put a burdon on both developments; xetex is the fast tex variant that uses libraries and has dvi as intermediate format, while luatex is the flexible (at the cost of speed and to some extend system integration) with a focus currently on pdf; each has its use etc etc; if we (luatex team) at some point decide to drop dvi (as taco mentioned) then we take that freedom (and might provide an api for others to cook up a superdvi backend) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Yannis Haralambous wrote:

Talking about DVI and PDF, I am the first one to be concerned about the programmed extinction of DVI because I have many jobs based on DVI post-processing (for example for marginal material, headers, and even for parallel texts between two pages). The advantage with DVI is that you can very easily now where you are located on the page so that you can extract text with positional information, and you can also easily insert new text (using pushes and pops).

i'm not sure if you refer to the same kind of positional info, but pdftex (and therefore luatex) hasn pdfsavepos cum suis in both pdf and dvi mode so you can store positions (lazy, i.e. write them to file in the backend) and then use them in a second pass (it't a rather old feature by now, which started as an experiment by Thanh and me and for some time was only available in pdf mode and in dvi mode we could use specials and the dvipos postprocessor; after a while it was also added to dvi mode) on the agenda is some more positional (runtime, in prerolls using boxes and getting pos into afterwards), a kind of pet project in parallel

And with the Mars format there is always hope that all that will be written in XML...

and xps .. although one might have problems then with packages resources (and files become large too) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Yannis Haralambous wrote:

1) I put tags in the DVI file which allow me to place marginal material at (exactly) the same height, during post-processing. In DVI it is easy to place a PUSH, make a skip, typeset the material and then POP back to the previous position so that the rest of the page remains unchanged. I guess this will be harder in PDF.

i do that in pdf by using pdfsavepos cum suis (just calculate the delta between several positions and then move the material)

2) I calculate the height of lines so that I can detect whether a page is shorter or taller than the normal. This is very useful to prevent accidents.

In a more elaborate project I split my DVI page into lines (lines of text, of translation, of footnotes of translation and of one or more critical apparati) and I re-assemble these lines into double pages, by letting the critical apparati flow under the footnotes, ad by keeping parallelism between text and translation. This works fine with DVI, maybe everyline can become a BT...ET section in PDF and I can get the same result.

that can be done by looping over the nodelist and manipulating hlist nodes Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------