March 2006 - ntg-pdftex - NTG Mailing Lists

possible bug with non-closing files
by Vladimir Volovich 21 Apr '06

21 Apr '06

Hi! i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? Best, v.

3 2

[ pdftex-Bugs-498 ] pdftex leaves included files opened thus overflowing the ulimit
by noreply＠sarovar.org 27 Mar '06

27 Mar '06

Bugs item #498, was opened at 2006-03-26 23:57 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106 Category: PDF inclusion Group: v1.30.0 Status: Closed Resolution: Works For Me Priority: 3 Submitted By: Vladimir Volovich (vvv) Assigned to: Martin Schröder (oneiros) Summary: pdftex leaves included files opened thus overflowing the ulimit Initial Comment: [re-sending through sarovar bug reporting form] i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? ---------------------------------------------------------------------- >Comment By: Martin Schröder (oneiros) Date: 2006-03-27 14:51 Message: Logged In: YES user_id=421 Hans, please try it on Windows -- if it doesn't fail there, I'd be surprised. IIRC Fabrice' fix was for the \immediate version which also ran out of filedescriptors some years ago. ---------------------------------------------------------------------- Comment By: Hans Hagen (hagen) Date: 2006-03-27 09:52 Message: Logged In: YES user_id=927 (this problem does not occur on windows; i remember that fabrice looked into it long ago) there is a relationship between open files and using \immediate and/or the page key and/or re-use; using immediate solves the open file problems but may lead to bigger files; i asume that opening a file twice is taken care of since there was a request from choff (tug conference china) with respect to splitting the 'get dimensions' code (also needed for dvi mode) from the inclusion code it may make sense to look into that direction [for instance, provide a primitive to determine the dimensions and then always delay inclusion of the objects] there is also the matter of efficiently embedding fonts in included files (sharing with parent document) so that's another pitfall keeping files open has to do with sharing objects so patching this code in order to overcome the open file limits may turn out to be tricky ---------------------------------------------------------------------- Comment By: Vladimir Volovich (vvv) Date: 2006-03-27 09:39 Message: Logged In: YES user_id=2086 please clarify why non-\immediate \pdfximage is still allowed to leave the included files opened. (i.e. why the current behavior is considered normal, when the files are left opened and the ulimit eventually overflows). ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-27 02:58 Message: Logged In: YES user_id=421 The "problem" here is that \pdfximage normally is not immediate. If you make it immediate by e.g. inserting \let\mypdfximage\pdfximage \def\pdfximage{\immediate\mypdfximage} everything works fine. So maybe pdfpages should work use an \immediate. We still need to fix the problem with the remaining broken file, but that is another bug. ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-27 00:43 Message: Logged In: YES user_id=421 I at least found why the resulting file is not deleted (i.e. where pdfTeX actually crashes): The callstack is thus: - pdftex.web:scan_image - writeimg.c:readimage - writeimg.c:checktypebyheader - kpathsea/xfopen.c:xfopen, which calls FATAL_PERROR So we should register our error handler through atexit (3) ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106

1 0

[ pdftex-Bugs-498 ] pdftex leaves included files opened thus overflowing the ulimit
by noreply＠sarovar.org 27 Mar '06

27 Mar '06

Bugs item #498, was opened at 2006-03-26 21:57 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106 Category: PDF inclusion Group: v1.30.0 Status: Closed Resolution: Works For Me Priority: 3 Submitted By: Vladimir Volovich (vvv) Assigned to: Martin Schröder (oneiros) Summary: pdftex leaves included files opened thus overflowing the ulimit Initial Comment: [re-sending through sarovar bug reporting form] i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? ---------------------------------------------------------------------- >Comment By: Hans Hagen (hagen) Date: 2006-03-27 07:52 Message: Logged In: YES user_id=927 (this problem does not occur on windows; i remember that fabrice looked into it long ago) there is a relationship between open files and using \immediate and/or the page key and/or re-use; using immediate solves the open file problems but may lead to bigger files; i asume that opening a file twice is taken care of since there was a request from choff (tug conference china) with respect to splitting the 'get dimensions' code (also needed for dvi mode) from the inclusion code it may make sense to look into that direction [for instance, provide a primitive to determine the dimensions and then always delay inclusion of the objects] there is also the matter of efficiently embedding fonts in included files (sharing with parent document) so that's another pitfall keeping files open has to do with sharing objects so patching this code in order to overcome the open file limits may turn out to be tricky ---------------------------------------------------------------------- Comment By: Vladimir Volovich (vvv) Date: 2006-03-27 07:39 Message: Logged In: YES user_id=2086 please clarify why non-\immediate \pdfximage is still allowed to leave the included files opened. (i.e. why the current behavior is considered normal, when the files are left opened and the ulimit eventually overflows). ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-27 00:58 Message: Logged In: YES user_id=421 The "problem" here is that \pdfximage normally is not immediate. If you make it immediate by e.g. inserting \let\mypdfximage\pdfximage \def\pdfximage{\immediate\mypdfximage} everything works fine. So maybe pdfpages should work use an \immediate. We still need to fix the problem with the remaining broken file, but that is another bug. ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-26 22:43 Message: Logged In: YES user_id=421 I at least found why the resulting file is not deleted (i.e. where pdfTeX actually crashes): The callstack is thus: - pdftex.web:scan_image - writeimg.c:readimage - writeimg.c:checktypebyheader - kpathsea/xfopen.c:xfopen, which calls FATAL_PERROR So we should register our error handler through atexit (3) ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106

1 0

[ pdftex-Bugs-498 ] pdftex leaves included files opened thus overflowing the ulimit
by noreply＠sarovar.org 27 Mar '06

27 Mar '06

Bugs item #498, was opened at 2006-03-27 01:57 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106 Category: PDF inclusion Group: v1.30.0 Status: Closed Resolution: Works For Me Priority: 3 Submitted By: Vladimir Volovich (vvv) Assigned to: Martin Schröder (oneiros) Summary: pdftex leaves included files opened thus overflowing the ulimit Initial Comment: [re-sending through sarovar bug reporting form] i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? ---------------------------------------------------------------------- >Comment By: Vladimir Volovich (vvv) Date: 2006-03-27 11:39 Message: Logged In: YES user_id=2086 please clarify why non-\immediate \pdfximage is still allowed to leave the included files opened. (i.e. why the current behavior is considered normal, when the files are left opened and the ulimit eventually overflows). ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-27 04:58 Message: Logged In: YES user_id=421 The "problem" here is that \pdfximage normally is not immediate. If you make it immediate by e.g. inserting \let\mypdfximage\pdfximage \def\pdfximage{\immediate\mypdfximage} everything works fine. So maybe pdfpages should work use an \immediate. We still need to fix the problem with the remaining broken file, but that is another bug. ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-27 02:43 Message: Logged In: YES user_id=421 I at least found why the resulting file is not deleted (i.e. where pdfTeX actually crashes): The callstack is thus: - pdftex.web:scan_image - writeimg.c:readimage - writeimg.c:checktypebyheader - kpathsea/xfopen.c:xfopen, which calls FATAL_PERROR So we should register our error handler through atexit (3) ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106

1 0

[ pdftex-Feature Requests-87 ] In case of fatal errors, delete the output file
by noreply＠sarovar.org 27 Mar '06

27 Mar '06

Feature Requests item #87, was opened at 2004-07-02 11:13 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=496&aid=87&group_id=106 Category: None Group: v1.21 Status: Open Resolution: None Priority: 5 Submitted By: Martin Schröder (oneiros) Assigned to: Martin Schröder (oneiros) Summary: In case of fatal errors, delete the output file Initial Comment: When pdfTeX currently encounters a fatal error, it closes it's output file, which leads to incomplete/corrupt files. pdfTeX should produce only valid files, and in case of fatal errors the output should be deleted. ---------------------------------------------------------------------- >Comment By: Martin Schröder (oneiros) Date: 2006-03-27 03:00 Message: Logged In: YES user_id=421 Currently we only handle exits through pdf_fail. But we also need to trap other exits; this must be done through atexit(3). ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2005-12-19 00:38 Message: Logged In: YES user_id=421 The attached patch unlinks the current file, if a fatal error happens. Currently this is done without any checks, and may be a security problem. ---------------------------------------------------------------------- Comment By: Taco Hoekwater (taco) Date: 2004-07-02 14:35 Message: Logged In: YES user_id=1608 When you truncate a file, the result is an empty file. From a user's viewpoint, that is not the same thing as a corrupt PDF. The distinction is small, but the advantage of an empty file is that it is not likely to be confused for a correct PDF. If it is possible, then deleting the PDF file is definately better, but file deletion doesn't always work. Truncation almost always does work, and it's only two syscalls extra, so why not? ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2004-07-02 14:21 Message: Logged In: YES user_id=421 Simply truncating a pdf still makes it invalid (because the file structure is incomplete). Producing a valid truncated output file in case of fatal errors will be tricky and not always possible. Deleting the corrupt output is much easier. ---------------------------------------------------------------------- Comment By: Taco Hoekwater (taco) Date: 2004-07-02 13:50 Message: Logged In: YES user_id=1608 If deleting the file doesnt work, truncation is probably still better than nothing. (?) ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=496&aid=87&group_id=106

1 0

[ pdftex-Bugs-498 ] pdftex leaves included files opened thus overflowing the ulimit
by noreply＠sarovar.org 27 Mar '06

27 Mar '06

Bugs item #498, was opened at 2006-03-26 23:57 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106 Category: PDF inclusion Group: v1.30.0 >Status: Closed >Resolution: Works For Me >Priority: 3 Submitted By: Vladimir Volovich (vvv) Assigned to: Martin Schröder (oneiros) Summary: pdftex leaves included files opened thus overflowing the ulimit Initial Comment: [re-sending through sarovar bug reporting form] i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? ---------------------------------------------------------------------- >Comment By: Martin Schröder (oneiros) Date: 2006-03-27 02:58 Message: Logged In: YES user_id=421 The "problem" here is that \pdfximage normally is not immediate. If you make it immediate by e.g. inserting \let\mypdfximage\pdfximage \def\pdfximage{\immediate\mypdfximage} everything works fine. So maybe pdfpages should work use an \immediate. We still need to fix the problem with the remaining broken file, but that is another bug. ---------------------------------------------------------------------- Comment By: Martin Schröder (oneiros) Date: 2006-03-27 00:43 Message: Logged In: YES user_id=421 I at least found why the resulting file is not deleted (i.e. where pdfTeX actually crashes): The callstack is thus: - pdftex.web:scan_image - writeimg.c:readimage - writeimg.c:checktypebyheader - kpathsea/xfopen.c:xfopen, which calls FATAL_PERROR So we should register our error handler through atexit (3) ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106

1 0

[ pdftex-Bugs-498 ] pdftex leaves included files opened thus overflowing the ulimit
by noreply＠sarovar.org 27 Mar '06

27 Mar '06

Bugs item #498, was opened at 2006-03-26 23:57 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106 Category: PDF inclusion >Group: v1.30.0 Status: Open >Resolution: Accepted >Priority: 7 Submitted By: Vladimir Volovich (vvv) Assigned to: Martin Schröder (oneiros) Summary: pdftex leaves included files opened thus overflowing the ulimit Initial Comment: [re-sending through sarovar bug reporting form] i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? ---------------------------------------------------------------------- >Comment By: Martin Schröder (oneiros) Date: 2006-03-27 00:43 Message: Logged In: YES user_id=421 I at least found why the resulting file is not deleted (i.e. where pdfTeX actually crashes): The callstack is thus: - pdftex.web:scan_image - writeimg.c:readimage - writeimg.c:checktypebyheader - kpathsea/xfopen.c:xfopen, which calls FATAL_PERROR So we should register our error handler through atexit (3) ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106

1 0

[ pdftex-Bugs-498 ] pdftex leaves included files opened thus overflowing the ulimit
by noreply＠sarovar.org 26 Mar '06

26 Mar '06

Bugs item #498, was opened at 2006-03-27 01:57 You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106 Category: PDF inclusion Group: None Status: Open Resolution: None Priority: 5 Submitted By: Vladimir Volovich (vvv) Assigned to: Martin Schröder (oneiros) Summary: pdftex leaves included files opened thus overflowing the ulimit Initial Comment: [re-sending through sarovar bug reporting form] i'm reporting a bug discovered by Tigran Aivazian: He got this message: pdflatex: ./1016.pdf: Too many open files by running pdflatex on this simple source: \documentclass[pdftex,a5paper]{book} \usepackage[papersize={8.5in,11in}]{geometry} \usepackage{pdfpages} \begin{document} \newcount\i \i=2 \loop \ifnum\i<1725 \includepdf[pages={-}]{\number\i.pdf}% \advance\i by1 \repeat \end{document} It appears that this is due to the "ulimit" on the number of simultaneously opened files which was 1024 by default. Increasing "ulimit" allowed to process the file. But it looks like a bug in pdftex: it seems that it keeps all the files which were \includepdf'ed opened, thus causing this limit to be reached. Instead, it seems, it should close the included files. WDYT? ---------------------------------------------------------------------- You can respond by visiting: http://sarovar.org/tracker/?func=detail&atid=493&aid=498&group_id=106

1 0

Changing the pdfTeX banner for Debian-specific changes
by Frank Küster 24 Mar '06

24 Mar '06

Hi, this might already be resolved - we're still dealing with pdfTeX 1.21a in teTeX -, but also not, and therefore interest you. Julian Gilbey <jdg(a)polya.uklinux.net> wrote: >> >> as requested by Martin Schr?der, the pdfTeX maintainer, I'd like to add >> >> something about libpoppler to the "pdeftex --version" output or the >> >> general banner, but I don't manage. According to him, it should be >> >> sufficient to change texk/web2c/pdfetexdir/pdfetexextra.in, but that >> >> doesn't have any effect here - probably we have such an ancient version >> >> (1.20 instead of 1.40 in which he is thinking...). >> > >> > Just modifying the #define BANNER=... line in that file does it for >> > me. >> >> Strange - I tried and got funny results, e.g. after changing >> >> #define BANNER "This is pdfeTeX, Version 3.141592-PDFTEX-VERSION-ETEX-VERSION" >> >> to >> >> #define BANNER "This is not pdfeTeX, Version 3.141592-PDFTEX-VERSION-ETEX-VERSION" >> >> it printed "not pdfeTeX..." instead of "This is pdfeTeX..." > > With the normal version, compare: > > burnside:~ $ pdfetex --version > pdfeTeX 3.141592-1.21a-2.2 (Web2C 7.5.4) > kpathsea version 3.5.4 > Copyright (C) 1997-2004 Peter Breitenlohner (eTeX)/Han The Thanh (pdfTeX). > Kpathsea is copyright (C) 1997-2004 Free Software Foundation, Inc. > [...] > > with: > > burnside:~ $ pdfetex > This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4) > ** > > It turns out that when producing the version string for --version in > lib/printversion.c, function printversionandexit, the leading prefix > "This is " is stripped off from the banner name. Furthermore, the > banner string is parsed like this: > > "This is pdfeTeX, Version 3.141592-PDFTEX-VERSION-ETEX-VERSION" > ^^^^^^^^------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > stripped name progversion > > where "This is " is stripped, the prog name goes up to the first > comma, and the progversion is everything after the final space; then > the first line of the --version output is: > > name progversion (Web2C-version) Therefore it is indeed hard to add information about local changes, which naturally would be at the end (after an additional space). > So if all is wanted is to tweak the banner line, there are two ways to > do it: > > (1) Hackish: define BANNER to be something like: > "This is pdfeTeX with Debian tweaks, Version 3.141592-PDFTEX-VERSION-ETEX-VERSION" I'll go that way, but I'd appreciate if in future versions it would be possible to just add a "local_changes_remark" at an appropriate place (either the end of the line, or an additional line. > (2) Actually modify the printversionandexit code in lib/printversion.c > to output something like: > > string debianversion = "(Debian version blah)"; > > len = prog_name_end - banner - sizeof ("This is") + sizeof(debianversion); > prog_name = (string)xmalloc (len + 1); > strncpy (prog_name, banner + sizeof ("This is"), len); > prog_name[len] = 0; > > /* The Web2c version string starts with a space. */ > printf ("%s %s%s %s\n", prog_name, prog_version, versionstring, debianversion); > > and a similar change in one of the change files. > > But the latter seems like hard work :/ Should I open a feature request? Regards, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

1 0

Creating identical PDF files with different pdfTeX runs
by Frank Küster 20 Mar '06

20 Mar '06

Hi, we (i.e. a couple of Debian developers, taking up old ideas of each individual and the tests in http://tug.org/svn/texlive/trunk/Master/support/tests/) are trying to implement some automated testing of pdf and dvi creation by pdftex. Possible applications are regression tests for the binary, for distributions (changed font setup etc.), or for package authors ("does the new version still cooperate with hyperref?"). For this, it would be great if it were possible to create identical pdf files in subsequent runs of pdfTeX. With \pdfinfo{/CreationDate (1980-09-09)} \year=1980 \month=9 \day=9 \time=10 \pdfcompresslevel0 I get pdf files that can be compared, but only after some grepping: egrep -a -v '/BaseFont|/FontName|/UniqueID|/ID|/CreationDate' $< > $@ With dvipdfmx (and perhaps with newer pdfTeX versions than the one in teTeX) /CMap or something like this needs to be excluded, too. Is it possible to achieve identical pdf files directly, by adding the proper commands, or would it be possible to add this feature? TIA, Frank -- Frank Küster Single Molecule Spectroscopy, Protein Folding @ Inst. f. Biochemie, Univ. Zürich Debian Developer (teTeX)

8 24