Runtime limitations on open files?
Hi, we are crashing with "too many open files" on PDFTeX from TeXlive. The object id where this happens is 513 or so, and it would be my guess that PDFTeX keeps image files open as long as it has not yet shipped them out. It would appear that the system limitations of win32 are much larger, so I suspect the runtime with which TeXlive2007 has been compiled. Does anybody have a good pointer to where I might look for further information? This is quite a bad roadblock for us. -- David Kastrup
"DK" == David Kastrup writes: DK> Does anybody have a good pointer to where I might look for DK> further information? This is quite a bad roadblock for us. http://www.ntg.nl/pipermail/ntg-pdftex/2006-March/002186.html Best, v.
Vladimir Volovich
"DK" == David Kastrup writes:
DK> Does anybody have a good pointer to where I might look for DK> further information? This is quite a bad roadblock for us.
http://www.ntg.nl/pipermail/ntg-pdftex/2006-March/002186.html
Interesting. "Closed" with "Worksforme" because an application accumulated file descriptors erroneously. The problem is that we actually _have_ this amount of outstanding images not yet shipped out. No programming error. Putting the single line _setmaxstdio(2048); into web2c startup does not seem like it would be an overly onerous complication, would it? -- David Kastrup
2007/8/17, David Kastrup
Interesting. "Closed" with "Worksforme" because an application accumulated file descriptors erroneously.
The problem is that we actually _have_ this amount of outstanding images not yet shipped out. No programming error.
Just curious: How? Do you have more than 512/1024 images on one page?
_setmaxstdio(2048);
into web2c startup does not seem like it would be an overly onerous complication, would it?
No. Best Martin
"Martin Schröder"
2007/8/17, David Kastrup
: Interesting. "Closed" with "Worksforme" because an application accumulated file descriptors erroneously.
The problem is that we actually _have_ this amount of outstanding images not yet shipped out. No programming error.
Just curious: How? Do you have more than 512/1024 images on one page?
Before we decide what to ship out. There is also the possibility of some box registers not being cleared out, but it is not easy to debug and most important to fix this in a reasonable time frame.
_setmaxstdio(2048);
into web2c startup does not seem like it would be an overly onerous complication, would it?
No.
I can't see any downsides to that, anyway. Might be nice to stop leaving files open indiscriminately, but this stopgap measure appears like an easy copout at the moment. -- David Kastrup
David Kastrup wrote:
I can't see any downsides to that, anyway. Might be nice to stop leaving files open indiscriminately, but this stopgap measure appears like an easy copout at the moment.
As stopgaps go, this is a pretty decent solution, I think. In general, simply closing the image files after the information discovery and then reopening them for inclusion won't do, as the file may be messed up externally in the time between those two events. On almost all systems, keeping an open file handle protects you against exactly that. Best wishes, Taco
On Fri, 17 Aug 2007, Taco Hoekwater wrote:
David Kastrup wrote:
I can't see any downsides to that, anyway. Might be nice to stop leaving files open indiscriminately, but this stopgap measure appears like an easy copout at the moment.
As stopgaps go, this is a pretty decent solution, I think. In general, simply closing the image files after the information discovery and then reopening them for inclusion won't do, as the file may be messed up externally in the time between those two events.
when the files are first opened, only very limited info is extracted (type, dimensions, density) and stored. The rest doesn't matter. So if one opens a file a second time for inclusion, only these stored parameters need to be checked against the ones in the file (just read parameters a 2nd time) for consistency, then nothing can be messed up, even if the image contents differs (the new image would be included). An error (warning?) would be given only if the dimension data don't match. Would it be critical for the image contents, if it has changed in the meantime? Is it needed to nail down the image by keeping the file open? Are images changing dynamically during a session? Maybe then it might be natural anyway to take the freshest one?
On almost all systems, keeping an open file handle protects you against exactly that.
Regards, Hartmut
Hartmut Henkel
On Fri, 17 Aug 2007, Taco Hoekwater wrote:
As stopgaps go, this is a pretty decent solution, I think. In general, simply closing the image files after the information discovery and then reopening them for inclusion won't do, as the file may be messed up externally in the time between those two events.
Would it be critical for the image contents, if it has changed in the meantime?
Probably.
Is it needed to nail down the image by keeping the file open?
Actually, this does not help: if another application has the file opened r/w, it can still clobber it while PDFTeX is running. In Posix terms, keeping the file open helps against the effects of creat or unlink, but not against open(..., O_TRUNC).
Are images changing dynamically during a session? Maybe then it might be natural anyway to take the freshest one?
On almost all systems, keeping an open file handle protects you against exactly that.
I don't think so. -- David Kastrup
2007/8/17, David Kastrup
"Martin Schröder"
writes: Just curious: How? Do you have more than 512/1024 images on one page?
Before we decide what to ship out. There is also the possibility of some box registers not being cleared out, but it is not easy to debug and most important to fix this in a reasonable time frame.
I'm curious because I know that with nonimmediate \pfximage the number of different pdfs on one page is essentially unlimited (at least 2080 work). Best Martin
David Kastrup
we are crashing with "too many open files" on PDFTeX from TeXlive. The object id where this happens is 513 or so, and it would be my guess that PDFTeX keeps image files open as long as it has not yet shipped them out.
It would appear that the system limitations of win32 are much larger, so I suspect the runtime with which TeXlive2007 has been compiled.
Does anybody have a good pointer to where I might look for further information? This is quite a bad roadblock for us.
Ok, I got this info on the Lua list: URL:http://msdn2.microsoft.com/en-us/library/6e3b887c(VS.80).aspx Given the other array sizes of TeX, bumping this to the maximum of 2048 seems like a no-brainer. Unfortunately, I don't have a compilation/crosscompilation environment available here. What is the fastest way to get a working binary? Is there a reasonably easy to set up compilation or crosscompilation environment? Thanks, -- David Kastrup
Hi,
we are crashing with "too many open files" on PDFTeX from TeXlive. The object id where this happens is 513 or so, and it would be my guess that PDFTeX keeps image files open as long as it has not yet shipped them out.
It would appear that the system limitations of win32 are much larger, so I suspect the runtime with which TeXlive2007 has been compiled. I remember years ago when I started with my conference wrapper scripts
David Kastrup wrote: that windows crashed around that number, and because I needed to access
1500 papers, Fabrice Popineau (in cooperation with Hans) provided a version with did it. Since then I never had any reason to complain, so may be you should ask Fabrice.
Cheers, Volker
participants (6)
-
David Kastrup
-
Hartmut Henkel
-
Martin Schröder
-
Taco Hoekwater
-
Vladimir Volovich
-
Volker RW Schaa