Hello, TL;DR: Could building LuaMetaTeX with the `-fPIC` flag for the Lua library have negative consequences? Is there a better way to be able to call external C functions from ConTeXt? I am working on a small project where it would be convenient to call functions written in C (possibly pre-compiled in a shared library) from ConTeXt. After looking at the Lua FFI, I found a way that seems to work (with ConTeXt LMTX 2024-08-16) but requires a small tweak to the cmake file used for building the Lua library. I would be grateful to have the opinion of people more knowledgeable than I am about whether this change could have negative consequences and whether there is a more straightforward way to call external C code. What I am trying to achieve is: * Compile some C code in a shared library (for instance, it may contain a numerical solver for some partial differential equation). * Have this code run when compiling a document with ConTeXt. (For instance, solving the differential equation with parameters defined in the .tex file and retrieving the results to plot a figure with MetaPost.) One easy way to do that is: * Write wrappers for the functions that will need to be called from ConTeXt using the Lua C API. * Compile the code and wrappers in a shared library and linking to the static lua library used by LuaMetaTeX (liblua.a on Linux). * In the .tex file, import the new library using, e.g., ``` \startluacode mylib = require("mylib") -- additional code using the C functions \stopluacode ``` * Run ConTeXt with the `--permitloadlib` flag. However, the second step requires (at least on Linux) liblua.a to be compiled with the `-fPIC` flag. I could do that by adding ``` set_property(TARGET lua PROPERTY POSITION_INDEPENDENT_CODE ON) ``` at line 47 in lua.cmake. Is there any known negative effect of doing that (apart from a possible small performance drop)? (Just to be clear, this is not a feature request; I'm just wondering whether doing it on my side is fine.) On a related note, is there a standard way to call external C functions (either in a shared library or in a .c file) from ConTeXt LMTX? The SwigLib library described in https://www.pragma-ade.nl/general/manuals/swiglib-mkiv.pdf looks very promising (and maybe could be a way to achieve what I'm trying to do in a much cleaner way), but I (maybe for lack of trying hard enough) have not been able to make it work with ConTeXt LMTX so far. Best regards, Florent
On 14 Sep 2024, at 21:46, Florent Michel
wrote: TL;DR: Could building LuaMetaTeX with the `-fPIC` flag for the Lua library have negative consequences? Is there a better way to be able to call external C functions from ConTeXt?
This seems a bit drastic. Why can’t you achieve what you want to achieve by using other methods? For example you could setup a workflow whereby you pre-calculate your DE’s and format the results to include as input files. Or you could pre-calculate your DEs and plot them using one of any number of plotting programs and include the plots as TIFFs or PNGs. If you’re familiar with compiling C code using tools like `make` then why not use a makefile to control the production of your DE plots and your ConTeXt document to ensure that everything is consistent? If you absolutely must run the DE calculations as the ConTeXt document is processed then there is \executesystemcommand. — Bruce Horrocks Hampshire, UK
Thanks Bruce for your very helpful reply!
The reason for my question was indeed consistency, e.g. ensuring figure
captions stay up to date with what the figures show when changing
parameters. Thinking more about it, the second solution you mention seems
to be a better option, though - I can simply define the parameters in an
external file and read it from both the PDF solver and ConTeXt to ensure
consistency. Thanks for mentioning it!
Thank you also for mentioning \executesystemcommand, which I was not aware
of!
Best regards,
Florent
Le dim. 15 sept. 2024 à 21:31, Bruce Horrocks
On 14 Sep 2024, at 21:46, Florent Michel
wrote: TL;DR: Could building LuaMetaTeX with the `-fPIC` flag for the Lua library have negative consequences? Is there a better way to be able to call external C functions from ConTeXt?
This seems a bit drastic. Why can’t you achieve what you want to achieve by using other methods?
For example you could setup a workflow whereby you pre-calculate your DE’s and format the results to include as input files. Or you could pre-calculate your DEs and plot them using one of any number of plotting programs and include the plots as TIFFs or PNGs.
If you’re familiar with compiling C code using tools like `make` then why not use a makefile to control the production of your DE plots and your ConTeXt document to ensure that everything is consistent?
If you absolutely must run the DE calculations as the ConTeXt document is processed then there is \executesystemcommand.
— Bruce Horrocks Hampshire, UK
___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl webpage : https://www.pragma-ade.nl / https://context.aanhet.net (mirror) archive : https://github.com/contextgarden/context wiki : https://wiki.contextgarden.net
___________________________________________________________________________________
On 15 Sep 2024, at 22:11, Florent Michel
wrote: Thanks Bruce for your very helpful reply!
The reason for my question was indeed consistency, e.g. ensuring figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it!
Thank you also for mentioning \executesystemcommand, which I was not aware of!
Another option might be to use a marker such as % DE_figure_here param1 param2 param3 in your ConTeXt source and then use 'awk' or another Unix text pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place. Bonus marks if it can automatically generate the caption based on the parameters! But if not, the marker can always follow the \placefigure macro e.g. \placefigure {This is a DE showing something and something else} % DE_figure_here param1 param2 param3 so the awk code only needs to insert the image file name. Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image. HTH — Bruce Horrocks Hampshire, UK
Am 16.09.24 um 12:59 schrieb Bruce Horrocks:
On 15 Sep 2024, at 22:11, Florent Michel
wrote: Thanks Bruce for your very helpful reply!
The reason for my question was indeed consistency, e.g. ensuring figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it!
Thank you also for mentioning \executesystemcommand, which I was not aware of!
Another option might be to use a marker such as
% DE_figure_here param1 param2 param3
in your ConTeXt source and then use 'awk' or another Unix text pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place.
Bonus marks if it can automatically generate the caption based on the parameters! But if not, the marker can always follow the \placefigure macro e.g.
\placefigure {This is a DE showing something and something else} % DE_figure_here param1 param2 param3
so the awk code only needs to insert the image file name.
Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image.
That sounds needlessly convoluted to me. I’d use Aditya’s filter module to call external programs, it also does caching, i.e. if the parameters don’t change, it won’t waste processing time. https://github.com/adityam/filter Hraban
On 16 Sep 2024, at 12:15, Henning Hraban Ramm
Am 16.09.24 um 12:59 schrieb Bruce Horrocks:
On 15 Sep 2024, at 22:11, Florent Michel
wrote: Thanks Bruce for your very helpful reply!
The reason for my question was indeed consistency, e.g. ensuring figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it!
Thank you also for mentioning \executesystemcommand, which I was not aware of! Another option might be to use a marker such as % DE_figure_here param1 param2 param3 in your ConTeXt source and then use 'awk' or another Unix text pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place. Bonus marks if it can automatically generate the caption based on the parameters! But if not, the marker can always follow the \placefigure macro e.g. \placefigure {This is a DE showing something and something else} % DE_figure_here param1 param2 param3 so the awk code only needs to insert the image file name. Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image.
That sounds needlessly convoluted to me. I’d use Aditya’s filter module to call external programs, it also does caching, i.e. if the parameters don’t change, it won’t waste processing time.
I think Florent already has a situation that can be described as convoluted. Whether this is more so only he can say. I didn’t suggest the filter module to start with because it filters a buffer through an external command which isn’t Florent’s use-case (which is just the ability to call an external command). However I see, buried deeply in the docs, under the heading "Special use case: \write18 with caching” there is the ability to call a single command which would work. Someone with better knowledge of the macro would need to explain how to adapt the ’size’ key to Florent’s usage to enable the caching. Regards, — Bruce Horrocks Hampshire, UK
Thanks again Bruce! I had not thought about the possibility of parsing
comments in the tex file from an external tool, but that sounds like a very
good option. I like the fact that it would allow the tex file to contain
all the information needed to keep track of what is represented without
having to duplicate the parameter values. And since my specific workflow is
indeed already convoluted it will not really add extra complexity.
Thanks also Henning for mentioning the filter module! I'll have a closer
look into how I could use it - and it will certainly come useful, either
for this project or a future one!
Best regards,
Florent
Le lun. 16 sept. 2024 à 13:08, Bruce Horrocks
On 16 Sep 2024, at 12:15, Henning Hraban Ramm
wrote: Am 16.09.24 um 12:59 schrieb Bruce Horrocks:
On 15 Sep 2024, at 22:11, Florent Michel
Thanks Bruce for your very helpful reply!
The reason for my question was indeed consistency, e.g. ensuring
Thank you also for mentioning \executesystemcommand, which I was not
aware of! Another option might be to use a marker such as % DE_figure_here param1 param2 param3 in your ConTeXt source and then use 'awk' or another Unix text
Bonus marks if it can automatically generate the caption based on the
wrote: figure captions stay up to date with what the figures show when changing parameters. Thinking more about it, the second solution you mention seems to be a better option, though - I can simply define the parameters in an external file and read it from both the PDF solver and ConTeXt to ensure consistency. Thanks for mentioning it! pre-processor to scan through for these, run the appropriate DE calculation, plot the graph and generate the appropriate \placefigure command to go in its place. parameters! But if not, the marker can always follow the \placefigure macro e.g.
\placefigure {This is a DE showing something and something else} % DE_figure_here param1 param2 param3 so the awk code only needs to insert the image file name. Note I’ve deliberately omitted the braces from the bit to be substituted so that you need to generate them in the awk. That way, if you forget to process, you’ll get an error not a missing image.
That sounds needlessly convoluted to me. I’d use Aditya’s filter module to call external programs, it also does caching, i.e. if the parameters don’t change, it won’t waste processing time.
I think Florent already has a situation that can be described as convoluted. Whether this is more so only he can say.
I didn’t suggest the filter module to start with because it filters a buffer through an external command which isn’t Florent’s use-case (which is just the ability to call an external command). However I see, buried deeply in the docs, under the heading "Special use case: \write18 with caching” there is the ability to call a single command which would work.
Someone with better knowledge of the macro would need to explain how to adapt the ’size’ key to Florent’s usage to enable the caching.
Regards, — Bruce Horrocks Hampshire, UK
___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl webpage : https://www.pragma-ade.nl / https://context.aanhet.net (mirror) archive : https://github.com/contextgarden/context wiki : https://wiki.contextgarden.net
___________________________________________________________________________________
participants (3)
-
Bruce Horrocks
-
Florent Michel
-
Henning Hraban Ramm