Am 25.01.2021 um 17:31 schrieb Hans Hagen :

...

If nobody has better advice, I’ll try to come up with a filter setup. That would mean we’d avoid the tikz module, every TikZ diagram will get written to a buffer of configurable name, and that (with preamble setup) will get run through LaTeX. I read pgf had some restrictions in plain TeX mode, but perhaps we can also use LuaTeX in plain mode to avoid the LaTeX dependency. In the following case the buffer only gets typeset when it changes:

Thank you (and Henri)! But we’d need to include the same setup header every time, so the filter module is probably really the best way. There’s just a problem with labels, I guess, if I go the filter-LaTeX or filter-TeX route. So filter-ConTeXt might make sense. I think I understand now why integrating MetaPost made a lot of sense. Hraban

Am 25.01.2021 um 18:58 schrieb Aditya Mahajan :

or some variant of that should also work. But the filter module does make it easier to store the results in another directory so that they are out of sight.

No, you need additional file system commands.

Here is a simple example:

Thank you! Meanwhile, looking at my LilyPond module, I got as far as: """ \startmodule[tikzfilter] \usemodule[filter] % docs see https://github.com/adityam/filter \def\TIKZTEMP{tikztemp} % name of folder for TikZ/buffer files \startbuffer[tikz::before] \environment env_world % contains font setup etc. \environment env_TikZ % contains TikZ settings \startpagefigure \starttikzpicture[thick,domain=-90:90]% TODO: make this configurable \stopbuffer \startbuffer[tikz::after] \stoptikzpicture \stoppagefigure \stopbuffer \define[1]\ParseTikZFile% #1 is the name of the output file {\ctxlua{thirddata.parsetikzfile("#1")}} \startluacode thirddata = thirddata or {} -- create temp folder if missing if not lfs.isdir("\TIKZTEMP") then lfs.mkdir("\TIKZTEMP") end function thirddata.parsetikzfile(name) -- name is like \TIKZTEMP/mainfile-temp-tikz-21.pdf logs.report("TIKZFILTER","name='" .. name .. "'") context("\\externalfigure[" .. name .. "]") end \stopluacode \defineexternalfilter[tikz][ cache=yes, purge=no, directory=\TIKZTEMP, bufferbefore={tikz::before}, bufferafter={tikz::after}, readcommand=\ParseTikZFile, output={\externalfilterbasefile.pdf}, % ConTeXt never writes results to other directories filtercommand={context --batchmode --once --autopdf=auto --result=\externalfilterbasefile\space \externalfilterinputfile\space && mv \externalfilterbasefile.pdf\space \TIKZTEMP/}, ] \stopmodule Hraban

Still working on that. I hope you can help me here, Aditya! I’ll add these issues to the tracker of t-filter, if it makes sense. Testing with: ConTeXt ver: 2021.01.24 16:07 LMTX fmt: 2021.1.25 The setup below works, but only with \starttikz ... \stoptikz so far, I don’t like the output file names, and using the working directory is tricky. (1) \processFILTERfile \processtikzfile{img/example.pgf} can’t find the file: t-filter > cached output file 'tikztemp/example1.pgf.pdf' missing. Rerunning filter t-filter > file 'tikztemp/example1.pgf.pdf' cannot be found t-filter > current filter : tikzThick t-filter > base file : example1.pgf t-filter > input file : img/example1.pgf t-filter > output file : tikztemp/example1.pgf.pdf ... system > 13: filename=img/example1.pgf foundname=img/example1.pgf fullname=.../img/example1.pgf usedmethod=direct Looks like ConTeXt can find the file, but the filter module cannot. --trackers=resolvers.schemes (or \enabletrackers) doesn’t change anything WRT log/output. Since this fails already, I don’t know if \processFILTERfile also uses bufferbefore/-after (it should). (2) working directory Since ConTeXt can’t put generated (temporary) files into a directory and --result also only renames afterwards, the directory key doesn’t help, I had to add "mv" and "cd" to my filtercommand. I didn’t check yet if caching works. (3) file names Using \processFILTERfile the original file name base is preserved, according to the log entry above. If I use \starttikz[name=example], I get "JOBNAME-temp-tikzThick-example.tmp". I’d like to get something like "COMPONENTNAME-##-example.tmp" or a path like "COMPONENTNAME/example_##.tmp". Is there a configuration hook in the module to change name generation? (4) subprocess parameters I’d like to forward some parameters from \starttikz to the temp buffer, e.g. \starttikz[name=example][mystyle=thick] (using the same syntax as \startsection with a second pair of brackets for custom parameters). And then have \starttikzpicture[\filterusersetupvariable{mystyle}] in the buffer. My workaround so far is a copy of the filter setup (tikz/tikzThick), since we have only a few different configurations. Best regards, Hraban """ % file: t-tikzfilter.tex \startmodule[tikzfilter] \usemodule[filter] % docs see https://github.com/adityam/filter \def\TIKZTEMP{tikztemp} % name of folder for TikZ/buffer files % normal setup \startbuffer[tikz::before] \environment env_world \environment env_TikZ \startTEXpage \starttikzpicture \stopbuffer % setup for graphics with thicker lines \startbuffer[tikzThick::before] \environment env_world \environment env_TikZ \startTEXpage \starttikzpicture[thick] % only difference \stopbuffer \startbuffer[tikz::after] \stoptikzpicture \stopTEXpage \stopbuffer \define[1]\ReadPDFFile% #1 is the name of the output file {\ctxlua{thirddata.readpdffile("#1")}} \startluacode thirddata = thirddata or {} local report_tikz = logs.reporter("tikzfilter") -- create temp folder if missing if not lfs.isdir("\TIKZTEMP") then lfs.mkdir("\TIKZTEMP") end function thirddata.readpdffile(name) -- name is like \TIKZTEMP/mainfile-temp-tikz-21.pdf report_tikz("name='%s'", name) % doesn’t work context("\\externalfigure[" .. name .. "]") end \stopluacode \defineexternalfilter[tikz][ cache=yes, % directory=\TIKZTEMP, % doesn’t work with ConTeXt bufferbefore={tikz::before}, bufferafter={tikz::after}, readcommand=\ReadPDFFile, output={\TIKZTEMP/\externalfilterbasefile.pdf}, filtercommand={mv \externalfilterbasefile.* \TIKZTEMP/ && cd \TIKZTEMP\space && context --batchmode --nodummy --once \externalfilterbasefile.tmp}, ] \defineexternalfilter[tikzThick][tikz][ bufferbefore={tikzThick::before}, ] \stopmodule """ """ % file: tikztest.tex \usemodule[t-tikzfilter] \usemodule[visual] \starttext \placefigure[margin,here][fig:example]{A box.} {\vskip1in\noindent% \starttikzThick[name=example0] \draw[fill=black!10] (0,4.75) rectangle (1.5,-4.75); \stoptikzThick } \fakewords{150}{200} \processtikzThickfile{tests/example1.pgf} \stoptext """

Am 30.01.2021 um 00:13 schrieb Aditya Mahajan :

For \processFILTER, t-filter does not even read the file... so it has to do with how the filtercommand is defined.

...

Since this fails already, I don’t know if \processFILTERfile also uses bufferbefore/-after (it should).

It does not. As I said, for processing external files, t-filter does not even read the file and just takes care of caching the result. It is assumed that the filter command can process the external file.

A pity. I didn’t understand that from the documentation. That means I need to setup different filters for internal and external graphics with the same content.

I can see if it is possible to active before and after filters while processing external files.

That would solve this problem for me and would make the behavior of filter commands more consistent. OTOH I can understand you don’t want to bloat your module.

Another option is to write the external files as valid tex file (i.e., with a \startTEXpage .. \stopTEXpage wrapper), and when defining the filtercommand, use something like `context --environment=env filename`, where env is the environment file with all the settings for fonts etc.

I don’t see this as an option, as we try to avoid code duplication over hundreds of graphics snippets. The \start/stop environment, as well as external files, should just contain the TikZ commands. All common setup/teardown code belongs in extra files/buffers.

...

(2) working directory

Since ConTeXt can’t put generated (temporary) files into a directory and --result also only renames afterwards, the directory key doesn’t help, I had to add "mv" and "cd" to my filtercommand. I didn’t check yet if caching works.

Caching should work if \definefilter is configured correctly.

Thanks.

From the point of view of t-filter module, this is a bug in the external program and you should contact upstream for a fix :-)

Yes. There were already several requests about ConTeXt using work/temp directories and changing the behavior of "--result".

Most (all?) other programs are happy to write a file in any directory (modulo write permissions, etc.).

I do not want to increase the complexity of the module to handle such corner cases.

I can understand that. I should probably let filter call an external script, or fork t-filter for my needs.

...

If I use \starttikz[name=example], I get "JOBNAME-temp-tikzThick-example.tmp". I’d like to get something like "COMPONENTNAME-##-example.tmp" or a path like "COMPONENTNAME/example_##.tmp". Is there a configuration hook in the module to change name generation?

No. I don't quite understand the concern. How do you plan to use a `name=value` option?

The current auto-generated file names are uncomfortably long, and I’d like to know which contains what. I’m not yet sure what would make the most sense for this project (at least sorting by chapter/component would), but I have an use case with LilyPond where I could save several LilyPond compile runs if I could influence the naming: I got ~500 songs in their own component files, and I use them in different songbooklets with the same layout. E.g. there’s one containing all the songs and several with a subject (e.g. winter & christmas, children, medieval & fantasy, some author...). They all use [name=\currentcomponent]. If I could avoid the JOBNAME part of the file name, all booklets would use the same note PDFs. Now, the temp directory contains literally thousands of files. Since this is just for fun, the additional space and compile time is just a bit annoying and doesn’t affect others, but I like to streamline my workflows...

...

(4) subprocess parameters

I’d like to forward some parameters from \starttikz to the temp buffer, e.g. \starttikz[name=example][mystyle=thick] (using the same syntax as \startsection with a second pair of brackets for custom parameters). And then have \starttikzpicture[\filterusersetupvariable{mystyle}] in the buffer.

My workaround so far is a copy of the filter setup (tikz/tikzThick), since we have only a few different configurations.

The code that I had posted already handles this, provided that the options start on the line after \startfilter:

Ah, I didn’t look closely enough. Thank you, that’s an usable workaround. Best, Hraban

Finally coming back to this.

Am 31.01.2021 um 07:39 schrieb Aditya Mahajan :

...

I’m not yet sure what would make the most sense for this project (at least sorting by chapter/component would), but I have an use case with LilyPond where I could save several LilyPond compile runs if I could influence the naming:

I got ~500 songs in their own component files, and I use them in different songbooklets with the same layout. E.g. there’s one containing all the songs and several with a subject (e.g. winter & christmas, children, medieval & fantasy, some author...). They all use [name=\currentcomponent]. If I could avoid the JOBNAME part of the file name, all booklets would use the same note PDFs. Now, the temp directory contains literally thousands of files. Since this is just for fun, the additional space and compile time is just a bit annoying and doesn’t affect others, but I like to streamline my workflows...

This is a relatively simple feature to add. I have pushed a new branch with this feature: https://github.com/adityam/filter/tree/prefix

I have added a new `prefix` option: you can use

\defineexternalfilter[....][prefix=whatever]

and then the filter module will write the temp name as `whateverfilter0.ext` `whateverfilter1.ext`. The default value of prefix is \jobname-temp- (note trailing dash), so there should be no difference in the default output, but now you can configure the prefix if you want.

That’s good, so if I set "prefix=,", I get "FILTER-COMPONENT.*", e.g. "lilypond-mysong-1.pdf" (LilyPond adds the number for each system). In the rare cases where component names collide or I need more than one filter environment per component, I can fix that manually.

I am also open to making filter name as part of the prefix, so that you can have the ability to remove it if you wish. But I am hesitant to do so because it can lead to hard to debug errors if there is a clash in the prefix of two different filters.

Not needed IMO.

I haven't tested it beyond simple tests to make sure that nothing breaks. You can copy t-filter.mkiv from https://raw.githubusercontent.com/adityam/filter/prefix/t-filter.mkiv to your working directory and test if works for your situation.

Works for me. Thank you very much! For the TikZ setup I’ll copy some of your code and sacrifice most of the configurability. Will report back... Hraban