Hans Hagen wrote:

- pdf itself is moving which may demand additional of different resources being added - the pdftex stream depends on for instance cm directives and font references and there has been changes in this area over time (improvements like collapsing, removing redundant code) - macro packages may change their implementations of annotations, color, graphics and such, which results in different object ordering, numbering and content - macro packages may add/support new features which in turn may result in differences between pdf files; - macro packages may improve/change/patch special things (hz metrics and such) - font resources may change (metrics are normally stable, but the rest may change)

I don't think that these arguments make such tests unuseful. If such changes occur, the tests will fail, and the known-good documents need to be regenerated and manually checked. However, most of the time this will *not* happen, and then the tests would be very helpful.

the best you can do is not to look at the pdf file, but to parse the log for errors, like overfull boxes which can be signals of old/new code doing weird things, missing fonts, map files, encodings and characters.

I don't think the log helps. The log files contain absolute paths, so they would need lots of replacements before you can even start comparing. They contain version information for the packages - but checking whether a new version gives identical results is one of our goals. And renamed files, or splitting a package into different input files loaded by the "master" file, would completely break when we would try to automatically parse the log file. Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

Hans Hagen wrote:

regression tests at the macro package (output) level is something else; afaik latex already has regression tests

Oh, very interesting - where are these?

; as said: analyzing the page stream or objects is non trivial, and even simple changes in pdftex or in annot/object/literal support will make a comparison nearly impossible

So maybe for pdfTeX development mainly the bitmap-based tests are useful; but for macro packages, based on a given pdfTeX version, I think the PDF based tests make sense, too. Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

Martin Schröder wrote:

On 2006-03-15 14:07:16 +0100, Frank Küster wrote:

...

we (i.e. a couple of Debian developers, taking up old ideas of each individual and the tests in http://tug.org/svn/texlive/trunk/Master/support/tests/) are trying to implement some automated testing of pdf and dvi creation by pdftex. Possible applications are regression tests for the binary, for distributions (changed font setup etc.), or for package authors ("does the new version still cooperate with hyperref?").

Good idea.

...

For this, it would be great if it were possible to create identical pdf files in subsequent runs of pdfTeX. With

This will only get you so far. If you want to compare PDFs, you should not test for identical files, but for identical output: Render the PDF to a bitmap (e.g. with ghostscript) and compare the generated bitmaps. Otherwise your test will fail whenever the output of pdfTeX is changed in any way.

That's a point; on the other hand, all I get then is glyphs on a page, but the information about the internal structure is lost: hyperlinks, character information (will a search for "fl" find the fl ligature, or will "ü" be pasted correctly?), etc.

Even then your tests will fail when we improve the typesetting of pdfTeX. :-}

I don't expect those tests, or rather the known-good documents, to be carved in stone. There will be chances on occasion which require manual checking of the new document. I guess it's best to have both kinds of test: Comparing the bitmaps gives information about the actual typesetting (and about the differences you wouldn't spot when checking by eye), comparing the pdf files gives additional information about the document structure.

A start would be setting the system date -- this would seed /ID and /CreationDate. Note that /ID also includes the filename.

The system date? You need to be root to do that, or is there a way to fake the system date in the local environment?

...

Is it possible to achieve identical pdf files directly, by adding the proper commands, or would it be possible to add this feature?

Currently it's impossible (and that's a feature :). Adding the feature can't be that hard, though.

I would very much appreciate this. Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

Martin Schröder wrote:

On 2006-03-15 15:06:43 +0100, Frank Küster wrote:

...

Martin Schröder wrote:

...

This will only get you so far. If you want to compare PDFs, you should not test for identical files, but for identical output: Render the PDF to a bitmap (e.g. with ghostscript) and compare the generated bitmaps. Otherwise your test will fail whenever the output of pdfTeX is changed in any way.

That's a point; on the other hand, all I get then is glyphs on a page, but the information about the internal structure is lost: hyperlinks, character information (will a search for "fl" find the fl ligature, or will "ü" be pasted correctly?), etc.

How far will you get with the logfile with a suitable combination of \tracingoutput etc.?

I don't know; I have never delved into font tracing commands. I'm also not sure that this is the best approach, because when we develop a testing framework, we want it to be extensible and easy to use, even for e.g. package authors. If they have a version of their package that produces a particular pdf feature (like a special internal hyperlink pointing to exactly the right place), they want to make sure that future versions don't loose that feature. For this, a suitable pdf file comparison is all that's needed. Parsing the log file might additionally be helpful to find out what's wrong, but it only works if the testing framework already considered this feature. Also, in case I'm one of the developers of this testing framework, I'd rather concentrate on implementing it cleanly, extensible, maintainable, etc., and not so much on learning about pdf internals I didn't need up to now, or pdfTeX-special tracing possibilities. Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

Sorry, I accidentally sent the previous mail before it was ready. There is another point I'd like to put on my wish list. When Knuth developed TeX, he obviously did not take into account that people want to pipe STDOUT into other programs. It would be nice to have an option "-q" or "--quiet" which instructs TeX to suppress any output except what is reqested implicitly by \write16, for instance. If you want to determine the number of pages in a particular pdf file, you can say pdftex '\pdfximage page 1 {file.pdf}\write16{\the\pdflastximagepages}\end' but you get a lot of more or less useless messages and have to filter the output somehow. Regards, Reinhard -- ---------------------------------------------------------------------------- Reinhard Kotucha Phone: +49-511-4592165 Marschnerstr. 25 D-30167 Hannover mailto:reinhard.kotucha@web.de ---------------------------------------------------------------------------- Microsoft isn't the answer. Microsoft is the question, and the answer is NO. ----------------------------------------------------------------------------

On Thu, 16 Mar 2006, Frank Küster wrote:

Reinhard Kotucha wrote:

...

<pagenumber> <md5sum of the bitmap file>

The bitmap files can be removed by the script when it is finished and standard UNIX tools can be used to examine the output files.

Particularly, diff(1) can be used efficiently. It will tell you the numbers of the pages which are different.

That's a very good suggestion, thanks!

this looks pretty fragile to me. Characters will end up in bitmaps with interpolated gray pixels, and so it depends not only on pdftex but also on any subtlety of the rendering engine. And if the md5sum doesn't match, you know nothing without the original file. Maybe some crosscorrelation between images with some given tolerance limit would be safer. Regards, Hartmut

On Thu, 16 Mar 2006, Frank Küster wrote:

Hartmut Henkel wrote:

Yes, we definitely need the original file.

...

Maybe some crosscorrelation between images with some given tolerance limit would be safer.

Any hints what I could use for such a purpose?

just found from Netpbm: NAME pnmpsnr - compute the difference between two portable anymaps SYNOPSIS pnmpsnr [pnmfile1] [pnmfile2] DESCRIPTION Reads two PBM, PGM, or PPM files, or PAM equivalents, as input. Prints the peak signal-to-noise ratio (PSNR) difference between the two images. This metric is typically used in image compression papers to rate the distortion between original and decoded image. And in ImageMagick there is a -compose operator with Difference and Multiply option. Regards, Hartmut

On 2006-03-16 20:08:02 +0100, Hartmut Henkel wrote:

And in ImageMagick there is a -compose operator with Difference and Multiply option.

This is exactly what's needed. I tried this (compare sample2e with sample2e & lm): - render the pdfs with gs to png the resulting pngs are not large (ca. 150k/page for sample2e) and can be tested with cmp/md5sum for identity - generate diff images with "composite -compose difference -negate" this eats memory (1G at 600dpi), but produces what's needed. A pointer: http://jpdfunit.sourceforge.net/ "JpdfUnit is a framework for testing a generated pdf document with the JUnit test framework so JPdfUnit is a high level api." Best Martin -- http://www.tm.oneiros.de

The Thanh Han wrote:

there are a few places where pdftex takes the current time into account to generate certain tags. Implementing a primitive to leave out date-related things is not difficult. Can you please give some further explain what it would be useful for?

1. Regression tests for package writers. I already have set a simple setup in one or two LaTeX packages of mine, and it was very helpful when I decided to make big changes to the implementation without changing the interface. I always wanted to generalize this, but never came to it. 2. Regression testing for TeX distributions or, e.g., rpm or deb packages of texlive or teTeX for some Linux distribution. When internals change (like rearranging TEXMF trees, putting fonts in other packages, changing dependencies, etc.), the documents should not change. This is actually the reason why we are now talking about this: There are a couple of Debian developers who invest a lot of effort in automated package tests, and one of them asked whether such thing was possible in teTeX. We started discussing, now I'm here. 3. Regression testing in pdfTeX: A change that only aims at e.g. performance enhancements should not alter the document. Many other changes of course will change it.

Also note that even when such a primitive is available, the chance you get the same output for a given input file is rather low. You need to ensure that *all* relevant files (like font-related stuff, macro, figure etc.) are also the same.

That means that such tests would have to be done in a "controlled environment". That's easy for application 2, and probably not too hard for 1. For 3, it's not trivial - I don't think you can run the tests after compiling, like other programs, e.g. perl, do, and hope they will succeed on any system. But if you recreate the known-good data on *your* system just before applying a problematic patch, and test afterwards, that should work. Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

On 2006-03-16 22:39:10 +0100, Hans Hagen wrote:

the second one could be an add=on for pdftex: a special log mode, where pdftex writes a file with all annotations (name, page, rectangle, maybe also the while dict) and a second one which lists all the used fonts, encoding files, map lines and glyphs (encoding subset)

I still think that this information can be found with \tracingoutput etc. Best Martin -- http://www.tm.oneiros.de

On 3/16/06, Hans Hagen wrote:

...

3. Regression testing in pdfTeX: A change that only aims at e.g. performance enhancements should not alter the document. Many other changes of course will change it.

i think that you must define different levels of similarity:

- pure text: in that case a bitmap as already discussed is needed - functionality (annotations and such): that need to take place at the pdf level, i.e. filtering resources and descriptions and compare them (e.g. annotation names, rectangles, etc) - font resources

If we focus on differences between pdf documents and forget for the moment that we are using pdftex, these levels apply to a much wider class of documents. I suspect it is too early to know what sort of differences will occur, so the first step is to find tools to analyze differences between PDF documents. Many people who have never heard of pdftex would like to be able to compare PDF's and analyze the contents, so if you create a good initial framework there will be lots of help. I've never used the commercial preflight tools, but based on the 4th hand reports I get (printer operator --> cust. service --> editor --> me) I gather that the commercial tools can provide pretty detailed reports (e.g., figure on page N has yellow in the "black" annotation text). I'm not sure that rasterizing is the best approach. What about a ghostscript device that generates various debugging summaries. Some things that would be useful: 1. list of objects with md5sums and page numbers where the object is used -- this would be used to focus in on areas where documents differ. If files differ on a small number of pages then comparing bitmaps might be useful and economical of resources. 2. human readable summary of a specified object We already have tools to pull out specified pages and extract text.

i can imagine that your test of text similarity is run with disabled interactive features

the second one could be an add=on for pdftex: a special log mode, where pdftex writes a file with all annotations (name, page, rectangle, maybe also the while dict) and a second one which lists all the used fonts, encoding files, map lines and glyphs (encoding subset)

It would be better to focus on things that can be used with other pdf generating applications. Many pdftex documents have pdf's from sources other than pdftex embedded. It would be worth thinking about ways to add metadata to pdf's to help the analyzers, but if we rely on the pdftex logs then we have to develop all the tools ourselves and if the difference is in an input pdf our tools may not work. If we produce a good framework I suspect the larger community of pdf users will become engaged and accomplish much more than the pdftex community can afford. -- George N. White III Head of St. Margarets Bay, Nova Scotia