regression testing Makefile
The contexttest repository now contains a Makefile to use for checking one context tree against another. There's a README alongside it but here's the quickstart svn export . /tmp/testdir cd /tmp/testdir make -k A=20070112 B=20070123 >& world.log (the -k is so it doesn't stop on the first error) It will 1. (if not already done) Download ConTeXt versions 2007.01.12 and 2007.01.23 from the museum, put each in its own subtree, make formats for each 2. Do the following for each test-NNN.tex file: a. Run it through 2007.01.12 to make test-NNN-20070112.pdf b. Run it through 2007.01.23 to make test-NNN-20070123.pdf c. Feed those pdfs to utils/compare-pdfs.sh to generate a bitmap diff d. Report differences found by compare-pdfs.sh (dB on each page) It was interesting running it. I haven't sorted through all the output yet, but a one point was that a few unicode fonts are a problem (maybe because I don't have them, but it makes texexec loop for a long time doing mktextfm and the like). More details to follow. Notes: If you want to view the image diffs as they are generated, set the viewer shell variable in compare-pdfs.sh (I often use xloadimage). I'd be very surprised if the Makefile worked on anything but Linux (uses features of bash + GNU make). I was pretty careful about re-entrancy, so the following works fine at least in my testing: make -k -j 4 A=20070112 B=20070123 -Sanjoy `A nation of slaves is always prepared to applaud the clemency of their master who, in the abuse of absolute power, does not proceed to the last extremes of injustice and oppression.' (Gibbon)
Sanjoy Mahajan wrote:
The contexttest repository now contains a Makefile to use for checking one context tree against another. There's a README alongside it but here's the quickstart
ah, brilliant
svn export . /tmp/testdir cd /tmp/testdir make -k A=20070112 B=20070123 >& world.log
(the -k is so it doesn't stop on the first error)
It will
1. (if not already done) Download ConTeXt versions 2007.01.12 and 2007.01.23 from the museum, put each in its own subtree, make formats for each
2. Do the following for each test-NNN.tex file:
a. Run it through 2007.01.12 to make test-NNN-20070112.pdf b. Run it through 2007.01.23 to make test-NNN-20070123.pdf c. Feed those pdfs to utils/compare-pdfs.sh to generate a bitmap diff d. Report differences found by compare-pdfs.sh (dB on each page)
It was interesting running it. I haven't sorted through all the output yet, but a one point was that a few unicode fonts are a problem (maybe because I don't have them, but it makes texexec loop for a long time doing mktextfm and the like). More details to follow.
ok, so we need to set up a procedure for that
Notes:
If you want to view the image diffs as they are generated, set the viewer shell variable in compare-pdfs.sh (I often use xloadimage).
I'd be very surprised if the Makefile worked on anything but Linux (uses features of bash + GNU make).
once everything is in place we can look into that, for the moment using linux only is ok
I was pretty careful about re-entrancy, so the following works fine at least in my testing:
make -k -j 4 A=20070112 B=20070123
-Sanjoy
`A nation of slaves is always prepared to applaud the clemency of their master who, in the abuse of absolute power, does not proceed to the last extremes of injustice and oppression.' (Gibbon) _______________________________________________ dev-context mailing list dev-context@ntg.nl http://www.ntg.nl/mailman/listinfo/dev-context
-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
a few unicode fonts are a problem (maybe
because I don't have them, but it makes texexec loop for a long time doing mktextfm and the like). More details to follow.
ok, so we need to set up a procedure for that
Yeah, the mktexnam processes spawned by pdftex won't easily die. In the python wrapper to handle timeouts and large logfiles, I had to do a bit of work: change the process group of the texexec process to a new one, let its children inherit that pgrp, and then do killpg on the whole bunch. Without the process group trick, it's not easy to find and kill the untamed processes. By the way, a previous email had the list of troublesome files (in terms of compliation, never mind the correctness!).
I'd be very surprised if the Makefile worked on anything but Linux (uses features of bash + GNU make).
once everything is in place we can look into that, for the moment using linux only is ok
It might work on MacOS X, since it has bash and probably has GNU make somewhere, perhaps as gmake (not sure if BSD make is similar enough to GNU make). -Sanjoy `Not all those who wander are lost.' (J.R.R. Tolkien)
Sanjoy Mahajan wrote:
Yeah, the mktexnam processes spawned by pdftex won't easily die. In the python wrapper to handle timeouts and large logfiles, I had to do a bit of work: change the process group of the texexec process to a new one, let its children inherit that pgrp, and then do killpg on the whole bunch. Without the process group trick, it's not easy to find and kill the untamed processes.
IIRC, you can inhibit all the auto-mktex scripts with a setting in texmf.cnf or mktex.cnf (or something). If these programs are needed, the test was almost certainly failed anyway. Best, Taco
IIRC, you can inhibit all the auto-mktex scripts with a setting in texmf.cnf or mktex.cnf (or something). If these programs are needed, the test was almost certainly failed anyway.
Good idea. Then I can just look at the missfont.log files. I'll do that in the next set of tests. For now, the auto-mktexing overruns the timeout, the wrapper script kills it, but the missfont.log has useful info (with one or two junk lines because the i/o buffers were not flushed). I'll also add --passon=-recorder so that input and output files can be checked. Although that change will improve only the recent tests (because --passon was was kaput until recently). -Sanjoy `Not all those who wander are lost.' (J.R.R. Tolkien)
Sanjoy Mahajan wrote:
a few unicode fonts are a problem (maybe
because I don't have them, but it makes texexec loop for a long time doing mktextfm and the like). More details to follow.
so its' missing tfm files? maybe no latin modern math installed? ok, so we need to set up a procedure for that
Yeah, the mktexnam processes spawned by pdftex won't easily die. In the python wrapper to handle timeouts and large logfiles, I had to do a bit of work: change the process group of the texexec process to a new one, let its children inherit that pgrp, and then do killpg on the whole bunch. Without the process group trick, it's not easy to find and kill the untamed processes.
By the way, a previous email had the list of troublesome files (in terms of compliation, never mind the correctness!).
I'd be very surprised if the Makefile worked on anything but Linux (uses features of bash + GNU make).
once everything is in place we can look into that, for the moment using linux only is ok
It might work on MacOS X, since it has bash and probably has GNU make somewhere, perhaps as gmake (not sure if BSD make is similar enough to GNU make).
-Sanjoy
`Not all those who wander are lost.' (J.R.R. Tolkien)
-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
a few unicode fonts are a problem so its' missing tfm files? maybe no latin modern math installed?
I don't think it's a lack of latin-modern math. I have the Debian lmodern-1.00-2 package, which should have the latin-modern math files too. In addition it seems to be a unicode-only problem. Here's one of the examples, from testing tex/context/base/unic-037.tex/test-001.tex : $ utils/what-logfiles-to-see.sh < world.log | xargs grep -l mktexnam tex/context/base/unic-037.tex/test-001-20070112.tlog tex/context/base/unic-037.tex/test-001-20070123.tlog metapost/context/base/mp-text.mp/test-001-20070112.tlog metapost/context/base/mp-text.mp/test-001-20070123.tlog The first piece of the pipe just gets the logfiles where texexec exited with an error. So those are the four of those logfiles that have mktexnam in them. Looking at the second log shows the kind of error I was getting: systems : begin file test-001-20070123 at line 19 (/tmp/a/ctxtrees/20070123/tex/context/base/unic-run.tex)kpathsea: Running mktextfm UnicodeRegular25 /usr/share/texmf/web2c/mktexnam: Could not map source abbreviation U for UnicodeRegular25. /usr/share/texmf/web2c/mktexnam: Need to update /usr/share/texmf-texlive/fonts/map/fontname/special.map? mktextfm: Running mf-nowin -progname=mf \mode:=ljfour; mag:=1; nonstopmode; input UnicodeRegular25 This is METAFONT, Version 2.71828 (Web2C 7.5.5) kpathsea: Running mktexmf UnicodeRegular25 ! I can't find file `UnicodeRegular25'. That error loops a bunch of times, then the alarm kills it after 20 seconds: utils/run-it.py: Timeout 20.0 sec reached [texexec --pdf --nonstopmode test-001-20070123.tex] -Sanjoy `A nation of slaves is always prepared to applaud the clemency of their master who, in the abuse of absolute power, does not proceed to the last extremes of injustice and oppression.' (Gibbon)
Sanjoy Mahajan wrote:
systems : begin file test-001-20070123 at line 19 (/tmp/a/ctxtrees/20070123/tex/context/base/unic-run.tex)kpathsea: Running mktextfm UnicodeRegular25 /usr/share/texmf/web2c/mktexnam: Could not map source abbreviation U for UnicodeRegular25. /usr/share/texmf/web2c/mktexnam: Need to update /usr/share/texmf-texlive/fonts/map/fontname/special.map? mktextfm: Running mf-nowin -progname=mf \mode:=ljfour; mag:=1; nonstopmode; input UnicodeRegular25 This is METAFONT, Version 2.71828 (Web2C 7.5.5) kpathsea: Running mktexmf UnicodeRegular25
! I can't find file `UnicodeRegular25'.
ok, this comes because UnicodeRegular is not mapped onto a real font; such unicode things assume a font split up in ranges, so, best ask the author of the test what was meant to be Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hans Hagen wrote:
Sanjoy Mahajan wrote:
systems : begin file test-001-20070123 at line 19 (/tmp/a/ctxtrees/20070123/tex/context/base/unic-run.tex)kpathsea: Running mktextfm UnicodeRegular25 /usr/share/texmf/web2c/mktexnam: Could not map source abbreviation U for UnicodeRegular25. /usr/share/texmf/web2c/mktexnam: Need to update /usr/share/texmf-texlive/fonts/map/fontname/special.map? mktextfm: Running mf-nowin -progname=mf \mode:=ljfour; mag:=1; nonstopmode; input UnicodeRegular25 This is METAFONT, Version 2.71828 (Web2C 7.5.5) kpathsea: Running mktexmf UnicodeRegular25
! I can't find file `UnicodeRegular25'.
ok, this comes because UnicodeRegular is not mapped onto a real font; such unicode things assume a font split up in ranges, so, best ask the author of the test what was meant to be
The author of the test may well have been me, I think I wrote a set for each of the (at that point in time) existing Unicode blocks. If the macros in unic-037 do not map to a real font and therefore raises an error, then the test 'works', in my view. Taco
Here are the missing font files from the regression testing (using the 2007.01.23 test tree, which is also the systemwide one here). I've also included example snippets from the logfiles. #!/bin/sh # find-missfonts.log for f in `find -name missfont.log` ; do echo == $f == grep '^mkt' $f | sort -u done $ utils/find-missfonts.sh == ./tex/context/base/unic-037.tex/missfont.log == mktextfm UnicodeRegular25 [from logfile] ... kpathsea: Running mktextfm UnicodeRegular25 /usr/share/texmf/web2c/mktexnam: Could not map source abbreviation U for UnicodeRegular25. /usr/share/texmf/web2c/mktexnam: Need to update /usr/share/texmf-texlive/fonts/map/fontname/special.map? ... == ./tex/context/base/s-chi-00.tex/missfont.log == [from logfile] ... (/tmp/a/ctxtrees/20070123/tex/context/base/s-chi-00.tex (/tmp/a/ctxtrees/20070123/tex/context/base/font-chi.tex loading : Context Font Macros / Chinese encoding : coding chi is loaded (/tmp/a/ctxtrees/20070123/tex/context/base/enco-chi.tex))) color : mpcmyk color space is supported color : mpspot color space is supported color : system rgb is global activated interaction : active (/tmp/a/ctxtrees/20070123/tex/context/base/sort-def.tex) (/tmp/a/ctxtrees/20070123/tex/context/base/sort-lan.tex) (/tmp/a/ctxtrees/20070123/tex/context/base/pdfr-def.tex)kpathsea: Running mktextfm uni-htsong-76 ... mktextfm uni-htsong-30 mktextfm uni-htsong-4e mktextfm uni-htsong-4f mktextfm uni-htsong-51 mktextfm uni-htsong-53 mktextfm uni-htsong-56 mktextfm uni-htsong-5d mktextfm uni-htsong-5e mktextfm uni-htsong-5f mktextfm uni-htsong-65 mktextfm uni-htsong-67 mktextfm uni-htsong-6c mktextfm uni-htsong-72 mktextfm uni-htsong-75 mktextfm uni-htsong-76 mktextfm uni-htsong-79 mktextfm uni-htsong-7a mktextfm uni-htsong-7b mktextfm uni-htsong-7c mktextfm uni-htsong-7e mktextfm uni-htsong-83 mktextfm uni-htsong-8d mktextfm uni-htsong-8f mktextfm uni-htsong-90 mktextfm uni-htsong-9a == ./tex/context/base/s-jap-00.tex/missfont.log == [from logfile] ... (/tmp/a/ctxtrees/20070123/tex/context/base/s-jap-00.tex (/tmp/a/ctxtrees/20070123/tex/context/base/font-jap.tex (/tmp/a/ctxtrees/20070123/tex/context/base/font-chi.tex loading : Context Font Macros / Chinese encoding : coding chi is loaded (/tmp/a/ctxtrees/20070123/tex/context/base/enco-chi.tex)) loading : Context Font Macros / Japanese )) color : mpcmyk color space is supported color : mpspot color space is supported color : system rgb is global activated interaction : active (/tmp/a/ctxtrees/20070123/tex/context/base/sort-def.tex) (/tmp/a/ctxtrees/20070123/tex/context/base/sort-lan.tex) (/tmp/a/ctxtrees/20070123/tex/context/base/pdfr-def.tex)kpathsea: Running mktextfm uni-cybercjk-76 mktextfm: Running mf-nowin -progname=mf \mode:=ljfour; mag:=1; nonstopmode; input uni-cybercjk-76 This is METAFONT, Version 2.71828 (Web2C 7.5.5) kpathsea: Running mktexmf uni-cybercjk-76 ! I can't find file `uni-cybercjk-76'. ... mktextfm uni-cybercjk-30 mktextfm uni-cybercjk-4f mktextfm uni-cybercjk-52 mktextfm uni-cybercjk-53 mktextfm uni-cybercjk-5b mktextfm uni-cybercjk-64 mktextfm uni-cybercjk-6a mktextfm uni-cybercjk-6b mktextfm uni-cybercjk-75 mktextfm uni-cybercjk-76 mktextfm uni-cybercjk-7a mktextfm uni-cybercjk-7b mktextfm uni-cybercjk-7f mktextfm uni-cybercjk-80 mktextfm uni-cybercjk-88 mktextfm uni-cybercjk-8a mktextfm uni-cybercjk-8b mktextfm uni-cybercjk-96 mktextfm uni-cybercjk-98 == ./metapost/context/base/mp-text.mp/missfont.log == [from logfile ... kpathsea: Running mktextfm ec-iwonar /usr/share/texmf/web2c/mktexnam: Could not map typeface abbreviation c- for ec-iwonar. /usr/share/texmf/web2c/mktexnam: Need to update /usr/share/texmf-texlive/fonts/map/fontname/special.map? mktextfm: Running mf-nowin -progname=mf \mode:=ljfour; mag:=1; nonstopmode; input ec-iwonar This is METAFONT, Version 2.71828 (Web2C 7.5.5) kpathsea: Running mktexmf ec-iwonar ! I can't find file `ec-iwonar'. ... mktextfm ec-iwonar mktextfm ex-iwonar mktextfm mi-iwonari mktextfm rm-iwonar mktextfm sy-iwonarz
participants (3)
-
Hans Hagen -
Sanjoy Mahajan -
Taco Hoekwater