Re: [NTG-context] Gnuplot module: Patching of Gnuplot binary still needed?
Dear Peter, On Mon, Nov 15, 2010 at 21:31, Peter Hedwig wrote:
Dear all,
I hope you don't mind that I reply to all but the ntg-context list (not subscribed, but feel free to forward the mail).
I included the list again, together with zipped attachments (te remain under the size limit).
Am Montag, 15. November 2010 schrieb Mojca Miklavec:
Apart from that ... I was working with Peter Hedwig on the ConTeXt part of his lua/TikZ terminal (that has already been included in gnuplot). We got it running, with some issues like general font loading scheme, but the work was never polished to the end (stalled at some point). We could revive that thread as well.
I am sorry, that it stalled a bit, but I worked on it secretly ;) But now I think it is time to release what is done so far to anyone who is willing to test it. Attached is my most recent dev. version of lua.trm and the gnuplot-tikz.lua script.
Wow! What a nice surprize!!!
Here is a brief list of changes to the gnuplot CVS version:
* support for ConTeXt and plain TeX
Hurray! Thanks a lot.
* removed dependencies for LaTeX packages other than TikZ * needs TikZ Version >= 2.0 * better support for saving terminal options with gnuplot 'save' command * fixed "overflow" on negative coordinates that are outside of the crop box * merged all recent patched from murphy-md and Ethan
Unfortunately it is the Context output that does not compile with the most recent Context version I installed from TeXLive 2010 and the http://tlcontrib.metatex.org/2010/ repository. But I remember it working with an older version so I hope this can be fixed easily ...
I'm almost sure that this has to do with the fact that TikZ is broken with respect to ConTeXt MKIV since 2009, but it has only been revealed publically since a recent update of TikZ (a month or two ago). In ConTeXt distribution we have frozen the updates of TikZ more than a year ago, waiting for the patch to be included upstream ... I have sent a working patch, but Till somehow left the project and I will try some luck again convincing the others to patch it.
To generate the style files and wrapper for the various TeX flavors just type
lua gnuplot-tikz.lua style
on the command line. The files generated should be
t-gnuplot-lua-tikz.tex (Context wrapper) gnuplot-lua-tikz.tex (plain TeX wrapper) gnuplot-lua-tikz.sty (LaTeX wrapper) gnuplot-lua-tikz-common.tex (commmon definitions)
and can be copied to the appropriate places.
lua gnuplot-tikz.lua help
prints the terminal help on stdout and lua gnuplot-tikz.lua termhelp > gnuplot-tikz.help generates the version to be included in gnuplot.
Feedback and patches welcome!
I will test and report back. Thanks again, Mojca
On Nov 17, 2010, at 11:47 , Mojca Miklavec wrote:
[...]
Here is a brief list of changes to the gnuplot CVS version:
* support for ConTeXt and plain TeX
Hurray! Thanks a lot.
* removed dependencies for LaTeX packages other than TikZ * needs TikZ Version >= 2.0 * better support for saving terminal options with gnuplot 'save' command * fixed "overflow" on negative coordinates that are outside of the crop box * merged all recent patched from murphy-md and Ethan
Unfortunately it is the Context output that does not compile with the most recent Context version I installed from TeXLive 2010 and the http://tlcontrib.metatex.org/2010/ repository. But I remember it working with an older version so I hope this can be fixed easily ...
Thanks, I just updated the CVS version of gnuplot with your files and gave it a try: It works with minimals!
[...]
Feedback and patches welcome!
No patches I'm afraid. But feedback goes here: 1) The label text of the following key element takes the color of the preceding key symbol. Minimal example: set terminal lua tikz fulldoc size 15cm,7cm context set output 'plot.tex' set xrange [ 0 : 13 ] f1(x)=sqrt(x) f2(x)=sqrt(x*0.8)-3 plot f1(x) lc rgb 'blue', f2(x) 2) Negative axis labels are set with hyphen instead of proper minus sign, which should be longer (see example above). 3) First I thought luatex hangs but then I noticed TikZ is awfully slow when drawing plots with large amounts of data points (the tikzpicture had about 3000 lines). I can speed this up by reducing the size by invoking plot with: plot "< gawk '(NR-1)%10 == 0 {print $0}' fileWithLotsOfData" using [...] I wonder if there is a smarter way to do this in gnuplot. 4) Is there a possibility to get rid of the vertical artifacts in filledcurves? Note: the artifacts are only visible when plotting from data files. set terminal lua tikz fulldoc size 15cm,7cm context set output 'plot.tex' plot "< gawk 'BEGIN {for (i = 1; i <= 100; i++) {print i, 0.7*i, 0.8*i+50}}'" using 1:2:3 with filledcurve lc rgb 'black' t 'filled curve' set xrange [ 0 : 13 ] plot abs(x) with filledcurve xy=2,5 Best wishes, Florian
On Fri, Nov 19, 2010 at 00:38, Florian Wobbe wrote:
Thanks, I just updated the CVS version of gnuplot with your files and gave it a try: It works with minimals!
[...]
Feedback and patches welcome!
No patches I'm afraid. But feedback goes here:
1) The label text of the following key element takes the color of the preceding key symbol. Minimal example:
set terminal lua tikz fulldoc size 15cm,7cm context set output 'plot.tex' set xrange [ 0 : 13 ] f1(x)=sqrt(x) f2(x)=sqrt(x*0.8)-3 plot f1(x) lc rgb 'blue', f2(x)
Confirmed. I had the problems with that in my terminal as well, but I think that it was a buglet in metapost back then; oh, good old memories ... :) :) :) I didn't check the details about this one.
2) Negative axis labels are set with hyphen instead of proper minus sign, which should be longer (see example above).
I'm not sure if this should really be fixed automatically or not. In order to fix this one would have to patch gnuplot core. I forgot the details how to do it (I can find it out), but you can configure the layout of labels. In particular you may enclose the number into math delimiters ($). Maybe something like set format y "$%.1f$" would work (you might want to change the size of left margin then). Doing any other kind of magic would be a bit dangerous. Keep in mind that plot 'x.dat' u 1:(2*$2) will also fail since TeX will interpret $ as math delimiter. But changing that behaviour would have other undesirable effects.
3) First I thought luatex hangs but then I noticed TikZ is awfully slow when drawing plots with large amounts of data points (the tikzpicture had about 3000 lines). I can speed this up by reducing the size by invoking plot with:
plot "< gawk '(NR-1)%10 == 0 {print $0}' fileWithLotsOfData" using [...]
I wonder if there is a smarter way to do this in gnuplot.
Using bitmap terminals. A while back I was dreaming about having mixed-mode with one terminal creating text labels and the other one (png for example) drawing the plot. But this needs some extra programming effort. You would stumble against the same problem when using PS or PDF terminal, maybe only at a slightly bigger number of points. What I did for my "theory of chaos" reports and the million-of-points plots in my thesis was to create an empty plot with ConTeXt terminal, create a borderless plot with PNG and then manually insert PNG into the generated ConTeXt output, which you could do with TikZ output as well. It requires more effort, but there is no other way to speed up vector graphics. They are simply not suitable for drawing millions of points on the same plot. (Gnuplot also has a keyword 'every' to reduce the number of points, but in my case I wanted to plot all of them.) Mojca
On Monday, November 22, 2010 07:01:07 am Mojca Miklavec wrote:
On Fri, Nov 19, 2010 at 00:38, Florian Wobbe wrote:
Thanks, I just updated the CVS version of gnuplot with your files and gave it a try: It works with minimals!
[...]
Feedback and patches welcome!
No patches I'm afraid. But feedback goes here:
1) The label text of the following key element takes the color of the preceding key symbol. Minimal example:
set terminal lua tikz fulldoc size 15cm,7cm context set output 'plot.tex' set xrange [ 0 : 13 ] f1(x)=sqrt(x) f2(x)=sqrt(x*0.8)-3 plot f1(x) lc rgb 'blue', f2(x)
Confirmed. I had the problems with that in my terminal as well, but I think that it was a buglet in metapost back then; oh, good old memories ... :) :) :) I didn't check the details about this one.
2) Negative axis labels are set with hyphen instead of proper minus sign, which should be longer (see example above).
I'm not sure if this should really be fixed automatically or not. In order to fix this one would have to patch gnuplot core.
I forgot the details how to do it (I can find it out), but you can configure the layout of labels. In particular you may enclose the number into math delimiters ($). Maybe something like set format y "$%.1f$" would work (you might want to change the size of left margin then). Doing any other kind of magic would be a bit dangerous. Keep in mind that plot 'x.dat' u 1:(2*$2) will also fail since TeX will interpret $ as math delimiter. But changing that behaviour would have other undesirable effects.
3) First I thought luatex hangs but then I noticed TikZ is awfully slow when drawing plots with large amounts of data points (the tikzpicture had about 3000 lines). I can speed this up by reducing the size by invoking plot with:
plot "< gawk '(NR-1)%10 == 0 {print $0}' fileWithLotsOfData" using [...]
I wonder if there is a smarter way to do this in gnuplot.
Using bitmap terminals. A while back I was dreaming about having mixed-mode with one terminal creating text labels and the other one (png for example) drawing the plot. But this needs some extra programming effort.
As I recall, I posted a patch the last time this discussion came up. It turns off text output from the png terminal without affecting any other graphical elements. My though was that it could be the basis of a pdflatex terminal class that works the same way as the current epslatex terminals. I can post it again, or upload it to SourceForge if it isn't there already. Ethan
You would stumble against the same problem when using PS or PDF terminal, maybe only at a slightly bigger number of points.
What I did for my "theory of chaos" reports and the million-of-points plots in my thesis was to create an empty plot with ConTeXt terminal, create a borderless plot with PNG and then manually insert PNG into the generated ConTeXt output, which you could do with TikZ output as well. It requires more effort, but there is no other way to speed up vector graphics. They are simply not suitable for drawing millions of points on the same plot.
(Gnuplot also has a keyword 'every' to reduce the number of points, but in my case I wanted to plot all of them.)
Mojca
On Nov 22, 2010, at 16:01 , Mojca Miklavec wrote:
[...]
2) Negative axis labels are set with hyphen instead of proper minus sign, which should be longer (see example above).
I'm not sure if this should really be fixed automatically or not. In order to fix this one would have to patch gnuplot core.
I forgot the details how to do it (I can find it out), but you can configure the layout of labels. In particular you may enclose the number into math delimiters ($). Maybe something like set format y "$%.1f$" would work (you might want to change the size of left margin then).
Hej Mojca, thank you for your answer! Well, that was surprisingly easy - I should have checked the manual first (I was just about to write an ugly hack into your context terminal driver :). Actually set format "$%g$" gives the default label format (only enclosed in $) for all axes.
Doing any other kind of magic would be a bit dangerous. Keep in mind that plot 'x.dat' u 1:(2*$2) will also fail since TeX will interpret $ as math delimiter. But changing that behaviour would have other undesirable effects.
3) First I thought luatex hangs but then I noticed TikZ is awfully slow when drawing plots with large amounts of data points (the tikzpicture had about 3000 lines). I can speed this up by reducing the size by invoking plot with:
plot "< gawk '(NR-1)%10 == 0 {print $0}' fileWithLotsOfData" using [...]
I wonder if there is a smarter way to do this in gnuplot.
[...] What I did for my "theory of chaos" reports and the million-of-points plots in my thesis was to create an empty plot with ConTeXt terminal, create a borderless plot with PNG and then manually insert PNG into the generated ConTeXt output, which you could do with TikZ output as well. It requires more effort, but there is no other way to speed up vector graphics. They are simply not suitable for drawing millions of points on the same plot.
I'd probably do the same in that case. But unlike you I have a lot less points - 5000 points instead of millions. Loot at this example, which is the slightly modified random demo script taken from the gnuplot page: set terminal context size 15.0cm,10.5cm standalone set output 'randomCtx.tex' set dummy t,y unset key set parametric set samples 5000, 5000 set style function dots set title "Lattice test for random numbers" set xlabel "rand(n) ->" set xrange [ 0.00000 : 1.00000 ] noreverse nowriteback set ylabel "rand(n + 1) ->" set yrange [ 0.00000 : 1.00000 ] noreverse nowriteback set zlabel "rand(n + 2) ->" set zrange [ 0.00000 : 1.00000 ] noreverse nowriteback plot rand(0), rand(0) set terminal lua tikz context size 15.0cm,10.5cm fulldoc set output 'randomLua.tex' replot Now look at the timing of processing the output files with context (final run only): context randomCtx.tex -> 3.150 seconds. context randomLua.tex -> 23.658 seconds. Further increasing the number of points to 15000 results in 8.978 vs. 62.878 seconds. Quite a big difference in processing time!
(Gnuplot also has a keyword 'every' to reduce the number of points, but in my case I wanted to plot all of them.)
True, but suppose you don't have evenly spaced data points. Then you don't want every other line but one point every 5 kilometres or something the like. Especially for line drawings it would be beneficial not to place every single point. Instead consecutive points should be skipped if they are close to each other (with regard to plot units) - it makes no sense to include points which you won't see anyway. This could be done by defining a grid with a certain (user defined) resolution and rounding the coordinates (plot units) of a line point to the nearest grid node. All consecutive line points falling on the same grid node should not be passed on to terminal drivers. The psxy utility of GMT (http://gmt.soest.hawaii.edu/) does this for instance. I am not aware of such a functionality within gnuplot but it would be a nice feature. Best wishes, Florian
On Mon, Nov 22, 2010 at 20:15, Florian Wobbe wrote:
Especially for line drawings it would be beneficial not to place every single point. Instead consecutive points should be skipped if they are close to each other (with regard to plot units) - it makes no sense to include points which you won't see anyway. This could be done by defining a grid with a certain (user defined) resolution and rounding the coordinates (plot units) of a line point to the nearest grid node. All consecutive line points falling on the same grid node should not be passed on to terminal drivers. The psxy utility of GMT (http://gmt.soest.hawaii.edu/) does this for instance. I am not aware of such a functionality within gnuplot but it would be a nice feature.
But this is an issue of Gnuplot, not something that a terminal writer is supposed to think of. One thing that I did implement in ConTeXt was that if I get instructions move_to(10,3) line_to(11,8) line_to(11,8) line_to(11,9) then one line_to(11,8) will be ignored (resolution is hard-coded in the terminal, but you could draw a smaller plot and then magnify it which would seemingly decrease resolution). But in most cases that wouldn't really help. About the timing difference between "set term context" and "set term lua context": it is quite possible that metapost library is much faster than TikZ which uses TeX-based macros. TeX might be slower in calculations than metapost (which uses C for low level calculations) and TikZ is not optimized for drawing ten-thousand segments. And Hans really tried hard to optimize the code. Processing the output with ConTeXt is also at least ten times slower that outputting straight to PS or PDF and if you try ConTeXt MKII it is about two times slower than MKIV. You probably didn't have a chance to try the first implementations of ConTeXt terminal for gnuplot. It needed 10 minutes for 13 ordinary plots and it ran out of memory if I tried to plot 14 of them!!! (The reason was the usage of "btex text etex" constructs which lead to ConTeXt runs inside metapost runs inside ConTeXt runs, all doubled, maybe the labels were compile even four times, usually with a separate instance of ConTeXt for each plot.) Now compare that speed difference and the enormous optimization that Hans implemented back in 2006 ... The slow speed of TikZ is not something that you could do much about. There might be some tiny room for optimization in gnuplot-lua-tikz-common.tex, but hardly any. LuaTeX-based TikZ could be reimplemented and that would bring most benefits, but since Till almost-quit the project and since it works out-of-the-box with pdfTeX macros it is highly unlikely to happen. But yes, it would be nice if also ConTeXt terminal would be included. It still doesn't support raw images, but most other features are present. Mojca
On Mon, Nov 22, 2010 at 22:19, Aditya Mahajan wrote:
On Mon, 22 Nov 2010, Mojca Miklavec wrote:
The slow speed of TikZ is not something that you could do much about.
Rewrite the pgf backend code in metapost.
But that requires Till (maybe Hans) or some other genius to work on it for some non-trivial amount time. And that is not likely to happen any time soon. Peter as the author of TikZ terminal for gnuplot is certanly not the one that could be adressed to speed up his terminal. Mojca
On Monday 22 November 2010 22:19:40 Aditya Mahajan wrote:
On Mon, 22 Nov 2010, Mojca Miklavec wrote:
The slow speed of TikZ is not something that you could do much about.
Rewrite the pgf backend code in metapost.
Or use metapost: \usemodule [graph] Alan
On Nov 22, 2010, at 22:03 , Mojca Miklavec wrote:
On Mon, Nov 22, 2010 at 20:15, Florian Wobbe wrote:
Especially for line drawings it would be beneficial not to place every single point. Instead consecutive points should be skipped if they are close to each other (with regard to plot units) - it makes no sense to include points which you won't see anyway. This could be done by defining a grid with a certain (user defined) resolution and rounding the coordinates (plot units) of a line point to the nearest grid node. All consecutive line points falling on the same grid node should not be passed on to terminal drivers. The psxy utility of GMT (http://gmt.soest.hawaii.edu/) does this for instance. I am not aware of such a functionality within gnuplot but it would be a nice feature.
But this is an issue of Gnuplot, not something that a terminal writer is supposed to think of.
Indeed, this was my point in the first place. It should be handled by gnuplot prior to handing the data over to the terminal driver.
But yes, it would be nice if also ConTeXt terminal would be included. It still doesn't support raw images, but most other features are present.
Right, this might help increase the awareness among academics. Also not everyone has the ability to build gnuplot from sources. Florian
On Mon, Nov 22, 2010 at 22:44, Florian Wobbe wrote:
On Nov 22, 2010, at 22:03 , Mojca Miklavec wrote:
On Mon, Nov 22, 2010 at 20:15, Florian Wobbe wrote:
Especially for line drawings it would be beneficial not to place every single point. Instead consecutive points should be skipped if they are close to each other (with regard to plot units) - it makes no sense to include points which you won't see anyway. This could be done by defining a grid with a certain (user defined) resolution and rounding the coordinates (plot units) of a line point to the nearest grid node. All consecutive line points falling on the same grid node should not be passed on to terminal drivers. The psxy utility of GMT (http://gmt.soest.hawaii.edu/) does this for instance. I am not aware of such a functionality within gnuplot but it would be a nice feature.
But this is an issue of Gnuplot, not something that a terminal writer is supposed to think of.
Indeed, this was my point in the first place. It should be handled by gnuplot prior to handing the data over to the terminal driver.
However you need to write to gnuplot-dev then. This mailing list cannot help you.
But yes, it would be nice if also ConTeXt terminal would be included. It still doesn't support raw images, but most other features are present.
Right, this might help increase the awareness among academics. Also not everyone has the ability to build gnuplot from sources.
But as I said ... there have to be others writing on gnuplot-dev to show interest to other gnuplot developers. I will continue to work with Peter to improve TikZ output and to include support into t-gnuplot.tex, but comparison such as the speed test demonstrated with those random points (sent to dev-gnuplot) may help. Mojca
participants (5)
-
Aditya Mahajan -
Alan BRASLAU -
Ethan Merritt -
Florian Wobbe -
Mojca Miklavec