distributed / parallel TeX?

Lars Huttar

15 Dec 2008 15 Dec '08

11:06 p.m.

Hello, We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop. So the question comes up, can TeX runs take advantage of parallelized or distributed processing? As many in the computer industries are aware, processor speed (clock rate) has plateaued; it is not going to continue rising at the rate it had been. Hence the common move to dual-core, quad-core, etc. machines. But applications in general cannot take advantage of multiple cores to speed their work unless they are architected to take advantage of them. We googled around a bit but were surprised not to find any real references to efforts at running TeX in parallel or on distributed networks or clusters. Wouldn't this be something that a lot of people would find useful? Or does everyone only use TeX for typesetting short papers? Sure, you can use manual tricks to speed up TeX processing. You can "comment out" sections of a document, or select them via modes. But then you have to remember where you did the commenting out, so you can reverse it. And you have no guarantees as to whether the inclusion/exclusion of section B will affect the layout of section C or not. Wouldn't it be nice if TeX (or a TeX wrapper, or macro package, or "typesetting system") could take care of this for you? What if you had a language -- or a few extensions to existing languages -- to give your typesetting engine hints or commands about where to split up your long document into fairly-independent chunks? What if you designed your document specifically to be typeset in independent, parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time? What if the typesetting system automated the stitching-back-together process of the chunks, gathering page reference info from each chunk to inform the next iteration of typesetting the other chunks? Has anyone been working on this already? It seems like it must have been discussed, but I don't know where to go to look for that discussion. Thanks, Lars

Show replies by date

Taco Hoekwater

16 Dec 16 Dec

8:08 a.m.

Hi Lars, Lars Huttar wrote:

...

Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

You should not need an average of 8 runs unless your document is ridiculously complex and I am curious what you are doing (but that is a different issue from what you are asking).

...

So the question comes up, can TeX runs take advantage of parallelized or distributed processing?

No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document. About the only safe things that can be threaded is the reading of resources (images and fonts) and, mostly because of the small gain, nobody has been working on that that I know of.

...

parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time?

if you are willing to promiss yourself that all chapters will be exactly 20 pages - no more, no less - they you can split the work off into separate job files yourself and take advantage of a whole server farm. If you can't ... Best wishes, Taco

Lars Huttar

6:13 p.m.

On 12/16/2008 2:08 AM, Taco Hoekwater wrote:

...

Hi Lars,

Lars Huttar wrote:

...
Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

You should not need an average of 8 runs unless your document is ridiculously complex and I am curious what you are doing (but that is a different issue from what you are asking).

...
So the question comes up, can TeX runs take advantage of parallelized or distributed processing?

No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document.

Thank you for your response. Certainly this is true in general and in the worst case, as things stand currently. But I don't think it has to be that way. The following could greatly mitigate that problem: - You could design your document *specifically* to make the parts independent, so that the true and authoritative way to typeset them is to typeset the parts independently. (You can do this part now without modifying TeX at all... you just have the various sections' .tex files input common "headers" / macro defs.) Then, by definition, a change in one section cannot affect another section (except for page numbers, and possibly left/right pages, q.v. below). - Most large works are divisible into chunks separated by page breaks and possibly page breaks that force a "recto". This greatly limits the effects that any section can have on another. The division ("chunking") of the whole document into fairly-separate parts could either be done manually, or if there are clear page breaks, automatically. - The remaining problem, as you noted, is how to fix page references from one section to another. Currently, TeX resolves forward references by doing a second (or third, ...) pass, which uses page information from the previous pass. The same technique could be used for resolving inter-chunk references and determining on what page each chunk should start. After one pass on of the independent chunks (ideally performed simultaneously by separate processing nodes), page information is sent from each node to a "coordinator" process. E.g. the node that processed section two tells the coordinator that chapter 11 starts 37 pages after the beginning of section two. The coordinator knows in what sequence the chunks are to be concatenated, thanks to a config file. It uses this information together with info from each of the nodes to build a table of what page each chunk should start on, and a table giving the absolute page number of each page reference. If pagination has changed, or is new, this info is sent back to the various nodes for another round of processing. If this distributed method of typesetting a document takes 1 additional iteration compared to doing it in series, but you get to split the document into say 5 roughly equal parts, you could presumably get the job done a lot quicker in spite of the extra iteration. This is a crude description but hopefully the idea is clear enough.

...

...
parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time?

if you are willing to promiss yourself that all chapters will be exactly 20 pages - no more, no less - they you can split the work off into separate job files yourself and take advantage of a whole server farm. If you can't ...

Yes, the splitting can be done manually now, and when the pain point gets high enough, we do some manual separate TeX runs. However, I'm thinking that for large works, there is enough gain to be had that it would be worth systematizing the splitting process and especially the recombining process, since the later is more error-prone. I think people would do it a lot more if there were automation support for it. I know we would. But then, maybe our situation of having a large book with dual columns and multipage tables is not common enough in the TeX world. Maybe others who are typesetting similar books just use commercial WYSIWYG typesetting tools, as we did in the previous edition of this book. Lars

Martin Schröder

9:31 p.m.

2008/12/16 Lars Huttar :

...

- You could design your document *specifically* to make the parts independent, so that the true and authoritative way to typeset them is to typeset the parts independently. (You can do this part now without modifying TeX at all... you just have the various sections' .tex files input common "headers" / macro defs.) Then, by definition, a change in one section cannot affect another section (except for page numbers, and possibly left/right pages, q.v. below).

True. Also with TeX if your paragraphs are independent of each other (i.e. they don't include references to others), they could the typeset in parallel and then handed over to the page builder.

...

- Most large works are divisible into chunks separated by page breaks and possibly page breaks that force a "recto". This greatly limits the effects that any section can have on another. The division ("chunking") of the whole document into fairly-separate parts could either be done manually, or if there are clear page breaks, automatically.

pdfTeX 1.50 knows about the page diversions (analogue to m4's divert and undivert). They have a lot of potential.

...

page number of each page reference. If pagination has changed, or is new, this info is sent back to the various nodes for another round of processing.

Hopefully stopping at some point. If you use something like varioref, you can end with infinite circles. :-) Best Martin

Lars Huttar

10:10 p.m.

On 12/16/2008 3:31 PM, Martin Schröder wrote:

...

2008/12/16 Lars Huttar :

...
- You could design your document *specifically* to make the parts independent, so that the true and authoritative way to typeset them is to typeset the parts independently. (You can do this part now without modifying TeX at all... you just have the various sections' .tex files input common "headers" / macro defs.) Then, by definition, a change in one section cannot affect another section (except for page numbers, and possibly left/right pages, q.v. below).

True. Also with TeX if your paragraphs are independent of each other (i.e. they don't include references to others), they could the typeset in parallel and then handed over to the page builder.

Good point... although doesn't the page optimization feed back into paragraph layout?

...

...
- Most large works are divisible into chunks separated by page breaks and possibly page breaks that force a "recto". This greatly limits the effects that any section can have on another. The division ("chunking") of the whole document into fairly-separate parts could either be done manually, or if there are clear page breaks, automatically.

pdfTeX 1.50 knows about the page diversions (analogue to m4's divert and undivert). They have a lot of potential.

Sounds useful. It's impressive if you can get "a correct table of contents in the first run" (says http://www.gust.org.pl/BachoTeX/2008/presentations/ms/handout.pdf)

...

...
page number of each page reference. If pagination has changed, or is new, this info is sent back to the various nodes for another round of processing.

Hopefully stopping at some point. If you use something like varioref, you can end with infinite circles. :-)

But this is just a problem of typesetting with TeX in general, not particular to parallel/distributed typesetting, right? IIRC, Knuth even says in the TeXbook that a really pathological case might never stabilize. Lars

Martin Schröder

10:17 p.m.

2008/12/16 Lars Huttar :

...

Good point... although doesn't the page optimization feed back into paragraph layout?

No. :-( Best Martin

Taco Hoekwater

17 Dec 17 Dec

8:47 a.m.

Martin Schröder wrote:

...

2008/12/16 Lars Huttar :

...
Good point... although doesn't the page optimization feed back into paragraph layout?

No. :-(

But from Lars' POV, that is good :) There are some interesting ideas in this discussion, but with the current state of the code base all of this will be exceedingly difficult (especially because of all the synchronisation issues). Unless someone wants to work on this idea him/herself (and that would be great, there are not nearly enough people working on TeX development!), you could remind me, say, two years from now? Best wishes, Taco

luigi scarso

16 Dec 16 Dec

9:15 p.m.

On Tue, Dec 16, 2008 at 9:08 AM, Taco Hoekwater wrote:

...

Hi Lars,

Lars Huttar wrote:

...
Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

You should not need an average of 8 runs unless your document is ridiculously complex and I am curious what you are doing (but that is a different issue from what you are asking).

So the question comes up, can TeX runs take advantage of parallelized or

...
distributed processing?

No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document.

Also 3.11 Theory of page breaking www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20*course*.pdf -- luigi

Lars Huttar

11:02 p.m.

On 12/16/2008 3:15 PM, luigi scarso wrote:

...

On Tue, Dec 16, 2008 at 9:08 AM, Taco Hoekwater mailto:taco@elvenkind.com> wrote:

Hi Lars,

Lars Huttar wrote:

...

So the question comes up, can TeX runs take advantage of parallelized or distributed processing?

No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document.

Also 3.11 Theory of page breaking www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20 http://www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20*course*.pdf

Certainly that is a tough problem (particularly in regard to laying out figures near the references to them). But again, if you can break down the document into chunks that are fairly independent of each other (and you almost always can for large texts), this problem seems no worse for distributed processing than for sequential processing. For example, the difficult part of laying out figures in Section 1 is confined to Section 1; it does not particularly interact with Section 2. References in Section 2 to Section 1 figures are going to be relatively distant from those figures regardless of page breaking decisions. Thus the difficult problem of page breaking is reduced to the sequential-processing case... still a hard problem, but one that can be attacked in chunks. Indeed, the greater amount of CPU time per page that is made available through distributed processing may mean that the algorithms can do a better job of page breaking than through sequential processing. Lars

Hans Hagen

17 Dec 17 Dec

8:22 a.m.

Lars Huttar wrote:

...

On 12/16/2008 3:15 PM, luigi scarso wrote:

...
On Tue, Dec 16, 2008 at 9:08 AM, Taco Hoekwater mailto:taco@elvenkind.com> wrote:

Hi Lars,

Lars Huttar wrote:

....

...
So the question comes up, can TeX runs take advantage of parallelized or distributed processing?

No. For the most part, this is because of another requisite: for applications to make good use of threads, they have to deal with a problem that can be parallelized well. And generally speaking, typesetting does not fall in this category. A seemingly small change on page 4 can easily affect each and every page right to the end of the document.

Also 3.11 Theory of page breaking www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20 http://www.cs.utk.edu/~eijkhout/594-LaTeX/handouts/TeX%20LaTeX%20*course*.pdf

Certainly that is a tough problem (particularly in regard to laying out figures near the references to them). But again, if you can break down the document into chunks that are fairly independent of each other (and you almost always can for large texts), this problem seems no worse for distributed processing than for sequential processing. For example, the difficult part of laying out figures in Section 1 is confined to Section 1; it does not particularly interact with Section 2. References in Section 2 to Section 1 figures are going to be relatively distant from those figures regardless of page breaking decisions. Thus the difficult problem of page breaking is reduced to the sequential-processing case... still a hard problem, but one that can be attacked in chunks. Indeed, the greater amount of CPU time per page that is made available through distributed processing may mean that the algorithms can do a better job of page breaking than through sequential processing.

you need to keep an eye on where tex spends its time on, and much is related to loading fonts, reading files, saving output, etc and with multiple threads one would have to coordinate that and make sure the time spent on it does not become larger overall for instance, in your document making these large tables takes a while only because bTABLE is not that fast, so when at some point i can redo part of such mechanisms in lua we might gain way more runtime than by running distributed Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

luigi scarso

8:53 a.m.

...

you need to keep an eye on where tex spends its time on, and much is related to loading fonts, reading files, saving output, etc and with multiple threads one would have to coordinate that and make sure the time spent on it does not become larger overall

for instance, in your document making these large tables takes a while only because bTABLE is not that fast, so when at some point i can redo part of such mechanisms in lua we might gain way more runtime than by running distributed

Hans

If you are under Linux you can try /dev/shm http://www.cyberciti.biz/tips/what-is-devshm-and-its-practical-usage.html -- luigi

Lars Huttar

1:50 p.m.

Thanks, everybody, for the discussion on running TeX distributed / in parallel. I am much educated about the state of the art. :-) Summary ... - There is plenty of optimization that normally can be done. If a ConTeXt run is taking a really long time, chances are that something is not being done according to the design. - For most (current) purposes, documents are small enough and ConTeXt is fast enough that the effort to automate distribution of typesetting runs may not be worthwhile. On the other hand, the usage of TeX might expand if greater throughput were available. - However, as things stand now, one can always divide documents up by hand, typeset the parts independently, and stitch them back together using tools such as divert/undivert. One can even design a document with the spec that the canonical typesetting process is to typeset the sections independently; then the sections can never affect each other, except for explicitly added inter-section effects like page reference updates. If you're not aware of MarkMail, it's a handy place to browse / search archives of mailing lists. This thread can be found at http://markmail.org/search/?q=ntg+context+distributed+parallel On 12/17/2008 2:47 AM, Taco Hoekwater wrote:

...

There are some interesting ideas in this discussion, but with the current state of the code base all of this will be exceedingly difficult (especially because of all the synchronisation issues).

Unless someone wants to work on this idea him/herself (and that would be great, there are not nearly enough people working on TeX development!), you could remind me, say, two years from now?

Sure. Thank you for your interest. I wasn't asking for someone to implement new features for this, though I would be happy to see it happen if it is worthwhile for the community. As Dr Dobb's says, "Single core systems are history" (http://www.ddj.com/hpc-high-performance-computing/207100560). Software that can take advantage of multiple cores (or threads, or distributed nodes) will continue to scale. Of course some effort, and often some adjustment, is necessary to enable programs to effectively use parallelism. I'll create a page at http://wiki.contextgarden.net/Parallel summarizing this discussion if that's OK. Regards, Lars

Hans Hagen

16 Dec 16 Dec

9:07 a.m.

Lars Huttar wrote:

...

Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

often there are ways to optimize the document style btw, in my laptop mk.tex which loads a bunch of real big fonts (esp punk slows down) does 10 pages/second (242 pages currently) so your setu opif probably not that efficient

...

So the question comes up, can TeX runs take advantage of parallelized or distributed processing? As many in the computer industries are aware, processor speed (clock rate) has plateaued; it is not going to continue rising at the rate it had been. Hence the common move to dual-core, quad-core, etc. machines. But applications in general cannot take advantage of multiple cores to speed their work unless they are architected to take advantage of them.

We googled around a bit but were surprised not to find any real references to efforts at running TeX in parallel or on distributed networks or clusters. Wouldn't this be something that a lot of people would find useful? Or does everyone only use TeX for typesetting short papers?

it all depends on what you process; for simple docs tex is rather fast: \starttext \setuplayout[page] \dorecurse{1000}{\null\page} \stoptext such a run takes in mkiv: 5.944 seconds, 1000 processed pages, 1000 shipped pages, 168.237 pages/second

...

Sure, you can use manual tricks to speed up TeX processing. You can "comment out" sections of a document, or select them via modes. But then you have to remember where you did the commenting out, so you can reverse it. And you have no guarantees as to whether the inclusion/exclusion of section B will affect the layout of section C or not.

often it's inefficient font calls that slow down the job (or big graphics that one can skip including in a first pass)

...

Wouldn't it be nice if TeX (or a TeX wrapper, or macro package, or "typesetting system") could take care of this for you?

mayne in the future we can do some parallization also, i'm thinking of 'one run with prerolls' but it has no high priority (maybe i'll do it when i need it for a project)

...

What if you had a language -- or a few extensions to existing languages -- to give your typesetting engine hints or commands about where to split up your long document into fairly-independent chunks? What if you designed your document specifically to be typeset in independent, parallel pieces so that you could guarantee that you would get the same result for section B whether or not you were typesetting the whole book at the same time?

there are quite some dependencies of pages on each other (take cross refs and how they might influence a next run)

...

What if the typesetting system automated the stitching-back-together process of the chunks, gathering page reference info from each chunk to inform the next iteration of typesetting the other chunks?

this is an option when you have to reprocess parts of the documents often

...

Has anyone been working on this already? It seems like it must have been discussed, but I don't know where to go to look for that discussion.

if i were you i'd first look into optimizing the style Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Aditya Mahajan

3:06 p.m.

On Tue, 16 Dec 2008, Hans Hagen wrote:

...

Lars Huttar wrote:

...
Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

often there are ways to optimize the document style

In Latex, there is a package called mylatex, which allows you to create a format consisting of your preamble. Then you can call latex with your own format and this speeds up things. This approach did not provide a significant speed improvement in latex for me, and I don't know whether it will do so in context. Hans and Taco, do you think that creating a personal format and possibly also dumping some font related info could provide a tradeoff between processing speed and disk space? Aditya

Hans Hagen

3:53 p.m.

Aditya Mahajan wrote:

...

On Tue, 16 Dec 2008, Hans Hagen wrote:

...
Lars Huttar wrote:

...
Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

often there are ways to optimize the document style

In Latex, there is a package called mylatex, which allows you to create a format consisting of your preamble. Then you can call latex with your own format and this speeds up things.

not on this document since the processing takes way more than the startup in jelle's doc it's the many many many local font definitions and the some 800 metapost graphics that are the culprit - define fonts beforehand - use unique mpgraphic when possible i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part (of course 1450 pages of columnsets and multipage bTABLE's also cost a bit of time)

...

This approach did not provide a significant speed improvement in latex for me, and I don't know whether it will do so in context. Hans and Taco, do you think that creating a personal format and possibly also dumping some font related info could provide a tradeoff between processing speed and disk space?

it depends, you might win maybe a second per run when on startup in cases where you have thousands of small runs but often using a small tree (like the minimals) pays of more Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Lars Huttar

5:25 p.m.

On 12/16/2008 9:53 AM, Hans Hagen wrote:

...

Aditya Mahajan wrote:

...
On Tue, 16 Dec 2008, Hans Hagen wrote:

...
Lars Huttar wrote:

...
Hello,

We've been using TeX to typeset a 1200-page book, and at that size, the time it takes to run becomes a big issue (especially with multiple passes... about 8 on average). It takes us anywhere from 80 minutes on our fastest machine, to 9 hours on our slowest laptop.

in jelle's doc it's the many many many local font definitions and the some 800 metapost graphics that are the culprit

We have close to 7000 mpgraphics, and they add about 15 minutes to the run time. But the run time was already quite long before we started using those.

...

- define fonts beforehand

OK, we will look into this. I'm sure Jelle knows about this but I'm a noob. I'm pretty sure we are not *loading* fonts every time, but maybe we're scaling fonts an unnecessary number of times. For example, we have the following macro, which we use thousands of times: \def\LN#1{{\switchtobodyfont[SansB,\LNfontsize]{#1}}} Would it help much to instead use \definefont[SansBLN][... at \LNfontsize] and then \def\LN#1{{\SansBLN{#1}}} ?

...

- use unique mpgraphic when possible

I would be interested to know if this is possible in our situation. Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model. The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this?

...

i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part

My last run gave about 0.25 pages per second on our fastest server, when taking into account multiple passes; that comes out to about 2 pps for --once.

...

(of course 1450 pages of columnsets and multipage bTABLE's also cost a bit of time)

Hans Hagen

5:37 p.m.

Lars Huttar wrote:

...

We have close to 7000 mpgraphics, and they add about 15 minutes to the run time.

most of them are the same so reusing them made sense

...

But the run time was already quite long before we started using those.

...
- define fonts beforehand

OK, we will look into this. I'm sure Jelle knows about this but I'm a noob. I'm pretty sure we are not *loading* fonts every time, but maybe we're scaling fonts an unnecessary number of times. For example, we have the following macro, which we use thousands of times: \def\LN#1{{\switchtobodyfont[SansB,\LNfontsize]{#1}}}

indeed this will define the scaled ones again and again (whole sets of them since you use a complete switch); internall tex reuses them but it only know so when they're defined

...

Would it help much to instead use \definefont[SansBLN][... at \LNfontsize] and then \def\LN#1{{\SansBLN{#1}}} ?

indeed: \definefont[SansBLN][... at \LNfontsize] but no extra { } needed: \def\LN#1{{\SansBLN#1}}

...

...
- use unique mpgraphic when possible

I would be interested to know if this is possible in our situation. Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model.

i sent jelle the patched files

...

The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this?

metafun manual: unique mp graphics

...

...
i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part

My last run gave about 0.25 pages per second on our fastest server, when taking into account multiple passes; that comes out to about 2 pps for --once.

the patched files do 5-10 pps on my laptop (was > 1 sec pp) so an improvement factor of at least 5 is possible there are probably other optimizations possible but i cannot spent too much time on it Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Lars Huttar

7:28 p.m.

On 12/16/2008 11:37 AM, Hans Hagen wrote:

...

Lars Huttar wrote:

...
We have close to 7000 mpgraphics, and they add about 15 minutes to the run time.

most of them are the same so reusing them made sense

...
But the run time was already quite long before we started using those.

...
- define fonts beforehand

OK, we will look into this. I'm sure Jelle knows about this but I'm a noob. I'm pretty sure we are not *loading* fonts every time, but maybe we're scaling fonts an unnecessary number of times. For example, we have the following macro, which we use thousands of times: \def\LN#1{{\switchtobodyfont[SansB,\LNfontsize]{#1}}}

indeed this will define the scaled ones again and again (whole sets of them since you use a complete switch); internall tex reuses them but it only know so when they're defined

...
Would it help much to instead use \definefont[SansBLN][... at \LNfontsize] and then \def\LN#1{{\SansBLN{#1}}} ?

indeed:

\definefont[SansBLN][... at \LNfontsize]

but no extra { } needed:

\def\LN#1{{\SansBLN#1}}

Thanks, we will try this. (Jelle, since you have worked with this a lot longer than I have, please stop me if you have concerns about my making this sort of change.)

...

...
...
- use unique mpgraphic when possible

I would be interested to know if this is possible in our situation. Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model.

i sent jelle the patched files

OK, I'll look to hear from him. Are these patches to support these kinds of borders on tables, thus no longer needing to use MPgraphics?

...

...
The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this?

metafun manual: unique mp graphics

Great... I converted our useMPgraphics to uniqueMPgraphics. This reduced our number of mpgraphics from 7000 to 800! Unfortunately the result doesn't look quite right... but since we may not need to use mpgraphics anyway thanks to your patches, I'll hold off on debugging the result.

...

...
...
i changes the definitions a bit and now get 5 pages per second on my laptop in luatex; xetex processes the pages a bit faster but spends way more time on the mp part

My last run gave about 0.25 pages per second on our fastest server, when taking into account multiple passes; that comes out to about 2 pps for --once.

the patched files do 5-10 pps on my laptop (was > 1 sec pp) so an improvement factor of at least 5 is possible

there are probably other optimizations possible but i cannot spent too much time on it

Thanks for all your help thus far. Lars

Yue Wang

17 Dec 17 Dec

2:57 a.m.

...

Unfortunately the result doesn't look quite right... but since we may not need to use mpgraphics anyway thanks to your patches, I'll hold off on debugging the result.

maybe you should pass the table width and hight to the mp graphics?

Lars Huttar

23 Dec 23 Dec

3:48 a.m.

New subject: error when using uniqueMPgraphics

On 12/16/2008 1:28 PM, Lars Huttar wrote [Re: [NTG-context] distributed / parallel TeX?]:

...

On 12/16/2008 11:37 AM, Hans Hagen wrote:

...
...
The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this?

Lars Huttar wrote: ... metafun manual: unique mp graphics

Great... I converted our useMPgraphics to uniqueMPgraphics. This reduced our number of mpgraphics from 7000 to 800!

Unfortunately the result doesn't look quite right... but since we may not need to use mpgraphics anyway thanks to your patches, I'll hold off on debugging the result.

When I use uniqueMPgraphics, things run faster, but I get errors and warnings. They look like this: ** WARNING ** Specified (image) object doesn't exist: MP::139 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(216.633, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::139" ** WARNING ** Error locating image file "MP::185" ** WARNING ** Specified (image) object doesn't exist: MP::185 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(250.44, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::185" ** WARNING ** Error locating image file "MP::141" ** WARNING ** Specified (image) object doesn't exist: MP::141 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(284.247, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::141" ** WARNING ** Error locating image file "MP::186" ** WARNING ** Specified (image) object doesn't exist: MP::186 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(316.111, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::186" ** WARNING ** Error locating image file "MP::143" ** WARNING ** Specified (image) object doesn't exist: MP::143 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(352.456, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::143" The tex looks like: \startMPinitializations numeric FrameDist, Inner, Middle, Outer ; FrameDist := 0.6; Middle := -0.2 ; Inner := Middle - FrameDist; Outer := Middle + FrameDist; \stopMPinitializations % 0 double sides \startuniqueMPgraphic{table:frame:lrtb} % all single draw OverlayBox enlarged Middle ; setbounds currentpicture to OverlayBox ; \stopuniqueMPgraphic % 1 double side \startuniqueMPgraphic{table:frame:Lrtb} % left double draw OverlayBox leftenlarged Outer topenlarged Middle bottomenlarged Middle rightenlarged Middle ; draw OverlayBox leftenlarged Inner topenlarged Middle bottomenlarged Middle rightenlarged Middle ; setbounds currentpicture to OverlayBox ; \stopuniqueMPgraphic ... % 0 double sides \defineoverlay[lrtb][\uniqueMPgraphic{table:frame:lrtb}] % 1 double side \defineoverlay[Lrtb][\uniqueMPgraphic{table:frame:Lrtb}] \defineoverlay[lRtb][\uniqueMPgraphic{table:frame:lRtb}] \defineoverlay[lrTb][\uniqueMPgraphic{table:frame:lrTb}] \defineoverlay[lrtB][\uniqueMPgraphic{table:frame:lrtB}] ... \def\setuptableone{\setupTABLE[frame=off,split=no,option=stretch,offset=0.85mm,distance=0mm,columndistance=0mm,leftmargindistance=0mm,rightmargindistance=0mm] \setupTABLE[r][each][height=0.6cm,align={middle,lohi}] \setupTABLE[2,3,4,5,6,7][4,5,6,7,8,9][align={flushright,lohi}] \setupTABLE[1][4,5,6,7,8,9][align={flushleft,lohi}] \setupTABLE[c][1][background={LRtb}] \setupTABLE[c][2,4][background={Lrtb}] ...} \bgroup \setuptableone \startTabTableHeader [nc=7,height=0.8cm,align={right,lohi}]\switchtobodyfont[SansB,\countryheadersize]{Table 1.} ...

Lars Huttar

5:33 a.m.

New subject: error when using uniqueMPgraphics

On 12/22/2008 9:48 PM, Lars Huttar wrote:

...

On 12/16/2008 1:28 PM, Lars Huttar wrote [Re: [NTG-context] distributed / parallel TeX?]:

...
On 12/16/2008 11:37 AM, Hans Hagen wrote:

...
...
The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}] Empirically, this seems to end up using one mpgraphic per table cell, hence our thousands of mpgraphics. I don't know why a new mpgraphic would be created for each cell. Can someone suggest a way to avoid this?

Lars Huttar wrote: ... metafun manual: unique mp graphics Great... I converted our useMPgraphics to uniqueMPgraphics. This reduced our number of mpgraphics from 7000 to 800!

Unfortunately the result doesn't look quite right... but since we may not need to use mpgraphics anyway thanks to your patches, I'll hold off on debugging the result.

When I use uniqueMPgraphics, things run faster, but I get errors and warnings. They look like this:

** WARNING ** Specified (image) object doesn't exist: MP::139 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(216.633, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::139" ** WARNING ** Error locating image file "MP::185" ** WARNING ** Specified (image) object doesn't exist: MP::185 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(250.44, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::185" ** WARNING ** Error locating image file "MP::141" ** WARNING ** Specified (image) object doesn't exist: MP::141 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(284.247, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::141" ** WARNING ** Error locating image file "MP::186" ** WARNING ** Specified (image) object doesn't exist: MP::186 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(316.111, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::186" ** WARNING ** Error locating image file "MP::143" ** WARNING ** Specified (image) object doesn't exist: MP::143 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(352.456, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::143"

The tex looks like:

\startMPinitializations numeric FrameDist, Inner, Middle, Outer ; FrameDist := 0.6; Middle := -0.2 ; Inner := Middle - FrameDist; Outer := Middle + FrameDist; \stopMPinitializations

% 0 double sides \startuniqueMPgraphic{table:frame:lrtb} % all single draw OverlayBox enlarged Middle ; setbounds currentpicture to OverlayBox ; \stopuniqueMPgraphic

% 1 double side \startuniqueMPgraphic{table:frame:Lrtb} % left double draw OverlayBox leftenlarged Outer topenlarged Middle bottomenlarged Middle rightenlarged Middle ; draw OverlayBox leftenlarged Inner topenlarged Middle bottomenlarged Middle rightenlarged Middle ; setbounds currentpicture to OverlayBox ; \stopuniqueMPgraphic ... % 0 double sides \defineoverlay[lrtb][\uniqueMPgraphic{table:frame:lrtb}] % 1 double side \defineoverlay[Lrtb][\uniqueMPgraphic{table:frame:Lrtb}] \defineoverlay[lRtb][\uniqueMPgraphic{table:frame:lRtb}] \defineoverlay[lrTb][\uniqueMPgraphic{table:frame:lrTb}] \defineoverlay[lrtB][\uniqueMPgraphic{table:frame:lrtB}]

...

\def\setuptableone{\setupTABLE[frame=off,split=no,option=stretch,offset=0.85mm,distance=0mm,columndistance=0mm,leftmargindistance=0mm,rightmargindistance=0mm] \setupTABLE[r][each][height=0.6cm,align={middle,lohi}] \setupTABLE[2,3,4,5,6,7][4,5,6,7,8,9][align={flushright,lohi}] \setupTABLE[1][4,5,6,7,8,9][align={flushleft,lohi}] \setupTABLE[c][1][background={LRtb}] \setupTABLE[c][2,4][background={Lrtb}] ...}

\bgroup \setuptableone \startTabTableHeader [nc=7,height=0.8cm,align={right,lohi}]\switchtobodyfont[SansB,\countryheadersize]{Table 1.} ...

Sorry, I left out a couple of things I meant to say... Basically, when we run with uniqueMPgraphic instead of useMPgraphic, we get a lot fewer mpgraphic files; but the tables we produce lack the outer border. And we get the above errors. Googling the errors leads us to dvipdfmx, e.g. http://tug.org/texlive/devsrc/Build/source/texk/dvipdfmx/src/spc_pdfm.c http://dvipdfmx.sourcearchive.com/documentation/1:20050831-5/pdfximage_8c-so... But I don't think dvipdfmx is the cause. Can anyone explain the error messages or suggest a fix? We would love to use uniqueMPgraphics for efficiency, but we can't if it gives the wrong result... Thanks, Lars

Wolfgang Schuster

7:30 a.m.

New subject: error when using uniqueMPgraphics

Am 23.12.2008 um 04:48 schrieb Lars Huttar:

...

When I use uniqueMPgraphics, things run faster, but I get errors and warnings. They look like this:

** WARNING ** Specified (image) object doesn't exist: MP::139 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(216.633, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::139" ** WARNING ** Error locating image file "MP::185" ** WARNING ** Specified (image) object doesn't exist: MP::185 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(250.44, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::185" ** WARNING ** Error locating image file "MP::141" ** WARNING ** Specified (image) object doesn't exist: MP::141 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(284.247, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::141" ** WARNING ** Error locating image file "MP::186" ** WARNING ** Specified (image) object doesn't exist: MP::186 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(316.111, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::186" ** WARNING ** Error locating image file "MP::143" ** WARNING ** Specified (image) object doesn't exist: MP::143 ** WARNING ** Interpreting special command uxobj (pdf:) failed. ** WARNING ** >> at page="32" position="(352.456, 268.835)" (in PDF) ** WARNING ** >> xxx "pdf:uxobj @MP::143"

The tex looks like:

\startMPinitializations numeric FrameDist, Inner, Middle, Outer ; FrameDist := 0.6; Middle := -0.2 ; Inner := Middle - FrameDist; Outer := Middle + FrameDist; \stopMPinitializations

% 0 double sides \startuniqueMPgraphic{table:frame:lrtb} % all single draw OverlayBox enlarged Middle ; setbounds currentpicture to OverlayBox ; \stopuniqueMPgraphic

% 1 double side \startuniqueMPgraphic{table:frame:Lrtb} % left double draw OverlayBox leftenlarged Outer topenlarged Middle bottomenlarged Middle rightenlarged Middle ; draw OverlayBox leftenlarged Inner topenlarged Middle bottomenlarged Middle rightenlarged Middle ; setbounds currentpicture to OverlayBox ; \stopuniqueMPgraphic ... % 0 double sides \defineoverlay[lrtb][\uniqueMPgraphic{table:frame:lrtb}] % 1 double side \defineoverlay[Lrtb][\uniqueMPgraphic{table:frame:Lrtb}] \defineoverlay[lRtb][\uniqueMPgraphic{table:frame:lRtb}] \defineoverlay[lrTb][\uniqueMPgraphic{table:frame:lrTb}] \defineoverlay[lrtB][\uniqueMPgraphic{table:frame:lrtB}]

...

\def \setuptableone {\setupTABLE [frame = off ,split = no ,option = stretch ,offset = 0.85mm ,distance = 0mm,columndistance=0mm,leftmargindistance=0mm,rightmargindistance=0mm] \setupTABLE[r][each][height=0.6cm,align={middle,lohi}] \setupTABLE[2,3,4,5,6,7][4,5,6,7,8,9][align={flushright,lohi}] \setupTABLE[1][4,5,6,7,8,9][align={flushleft,lohi}] \setupTABLE[c][1][background={LRtb}] \setupTABLE[c][2,4][background={Lrtb}] ...}

\bgroup \setuptableone \startTabTableHeader [nc=7,height=0.8cm,align={right,lohi}]\switchtobodyfont[SansB, \countryheadersize]{Table 1.} ...

Can you make a working example from your code. Wolfgang

Mojca Miklavec

16 Dec 16 Dec

6:40 p.m.

On Tue, Dec 16, 2008 at 6:25 PM, Lars Huttar wrote:

...

Most of our mpgraphics are due to wanting thick-and-thin or single-and-double borders on tables, which are not natively supported by the ConTeXt table model. The advice I received said to define each mpgraphic using \startuseMPgraphic (we have about 18 of these), associate them with overlays using \defineoverlay (again, we have 18), and then use them in table cells using statements like \setupTABLE[c][first][background={LRtb}]

In that case I would suggest you to use vrules/hrules to achieve the same. As long as you don't need to use complicated graphics (like Hans uses random frames) this should be doable and would speed up the process enormously. What's the compile time if you omit the MP graphics? Mojca

5843

Age (days ago)

5851

Last active (days ago)

List overview

Download

22 comments

9 participants

participants (9)

Aditya Mahajan
Hans Hagen
Lars Huttar
luigi scarso
Martin Schröder
Mojca Miklavec
Taco Hoekwater
Wolfgang Schuster
Yue Wang