Hello, I'm new to LuaTeX and Lua in general and I try to find tools to process datas stored in XML files. I want to do this with LaTeX (pdfLuaLaTeX in TeXLive 2008) but there's no tool. Contrary to LaTeX, ConTeXt does have abilities to work with XML files. Do you think it could be possible to have LuaTeX equipped with such abilities one day ? I read in the LuaTeX reference that LuaTeX comes with modules such as LuaZIP that is really a good thing. Do you think it would be possible to have something like the LuaXML module added to luatex too ? Jack
TeXWorld wrote:
Hello,
I'm new to LuaTeX and Lua in general and I try to find tools to process datas stored in XML files. I want to do this with LaTeX (pdfLuaLaTeX in TeXLive 2008) but there's no tool. Contrary to LaTeX, ConTeXt does have abilities to work with XML files. Do you think it could be possible to have LuaTeX equipped with such abilities one day ? I read in the LuaTeX reference that LuaTeX comes with modules such as LuaZIP that is really a good thing. Do you think it would be possible to have something like the LuaXML module added to luatex too ?
this is not so much related to luatex (typesetting engine) but more to the input which can be anything (tex, database, xml, whatever); context does not use libraries but does xml in pure lua adding all kind of 'modules' to lua will make it way too big and unmaintainable (apart from philosophical debates about which of the many variants to choose) in the future there will be an option to load libraries (once we've sorted out the best way to deal with it in a secure way in the tex tree etc) but even then maintainance of libraries is not part of luatex itself (the exceptions are the libs we really need for typesetting, like fontforge which are also adapted to luatex and therefore part of the kernel; luazip and luasocket somehow fit in the core but database connectivity, xml handling etc is way off) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hans Hagen wrote:
this is not so much related to luatex (typesetting engine) but more to the input which can be anything (tex, database, xml, whatever); context does not use libraries but does xml in pure lua
Meanwhile, there is a pure perl XML parser in the lua wiki: http://lua-users.org/wiki/LuaXml Look for "Classic Lua-only version", then "updated for 5.1" Best wishes, Taco
Hans Hagen a écrit :
TeXWorld wrote:
Hello,
I'm new to LuaTeX and Lua in general and I try to find tools to process datas stored in XML files. I want to do this with LaTeX (pdfLuaLaTeX in TeXLive 2008) but there's no tool. Contrary to LaTeX, ConTeXt does have abilities to work with XML files. Do you think it could be possible to have LuaTeX equipped with such abilities one day ? I read in the LuaTeX reference that LuaTeX comes with modules such as LuaZIP that is really a good thing. Do you think it would be possible to have something like the LuaXML module added to luatex too ?
this is not so much related to luatex (typesetting engine) but more to the input which can be anything (tex, database, xml, whatever); context does not use libraries but does xml in pure lua
Hello Hans, hello everybody I did not answer since Taco gave me a solution...
adding all kind of 'modules' to lua will make it way too big and unmaintainable (apart from philosophical debates about which of the many variants to choose)
Yes of course
in the future there will be an option to load libraries (once we've sorted out the best way to deal with it in a secure way in the tex tree etc) I was recently thinking about that and that's really good news. I have several questions :
-when do you think this option could be available il LuaTeX ? This year ? -who will decide which lua modules will be available for/used with LuaTeX ? Users ? -could it be possible to install and remove any of them ? Is it the kind of thing that could be done with the TeX distribution (I'm thinking of tlmgr in TeXLive for example)
but even then maintainance of libraries is not part of luatex itself (the exceptions are the libs we really need for typesetting, like fontforge which are also adapted to luatex and therefore part of the kernel; luazip and luasocket somehow fit in the core but database connectivity, xml handling etc is way off)
TeXWorld wrote:
Hans Hagen a écrit :
-when do you think this option could be available il LuaTeX ? This year ?
don't expect this too soon; there will be a code freeze for texlive (0.40) and later this year (around eurotex) version 0.50 (which a couple of pending issues resolved) at this moment we aim at 2012 for version 1.0, which will have stable interfaces and functionality (until then every aspect can change)
-who will decide which lua modules will be available for/used with LuaTeX ? Users ?
indeed, luatex is just a core with a few modules built in, the ones that make most sense and are not debatable (fontforge, a few small helpers, and basic io, some of them are patched) given that we have the ability to load modules a few years from now, then maintaining extra modules is not part of the project and up to the users or distributers; for instance there are many ways to deal with xml and it makes no sense to enforce methods; the same is actually true for more aspects of luatex ... we provide the basic system and how it's used is up to users and macro packages as all do it their own way
-could it be possible to install and remove any of them ? Is it the kind of thing that could be done with the TeX distribution (I'm thinking of tlmgr in TeXLive for example)
i don't know; things like that involve compiling, maybe other dependencies on libraries, installation mess, distributione policies, etc. as said: we will make the luatex kernal as small as possible, and any prescribed dependencies make it unstable hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
TeXWorld wrote:
in the future there will be an option to load libraries (once we've sorted out the best way to deal with it in a secure way in the tex tree etc) I was recently thinking about that and that's really good news. I have several questions :
We will almost certainly (but nothing is 100% yet) not be shipping lua libraries as part of the luatex distribution. Dynamic loading of libraries can not be guaranteed to work everywhere, and we want the core funtionality available even on platforms that don't support it, so the core libraries will remain built-in, as they are now. So far, there are no plans (nor infrastructure) to ship extra lua libraries.
-when do you think this option could be available il LuaTeX ? This year ?
Not before texlive 2009, but probably not long after either.
-who will decide which lua modules will be available for/used with LuaTeX ? Users ? -could it be possible to install and remove any of them ? Is it the kind of thing that could be done with the TeX distribution (I'm thinking of tlmgr in TeXLive for example)
If we are assuming for the moment that luatex itself doesn't ship those libraries, that would automaticlly make it a problem for the distributions and/or individual users. But let's get back to this later (end of summer). Best wishes, Taco
Taco Hoekwater a écrit :
TeXWorld wrote:
in the future there will be an option to load libraries (once we've sorted out the best way to deal with it in a secure way in the tex tree etc)
I was recently thinking about that and that's really good news. I have several questions :
We will almost certainly (but nothing is 100% yet) not be shipping lua libraries as part of the luatex distribution.
Dynamic loading of libraries can not be guaranteed to work everywhere, and we want the core funtionality available even on platforms that don't support it, so the core libraries will remain built-in, as they are now. So far, there are no plans (nor infrastructure) to ship extra lua libraries.
-when do you think this option could be available il LuaTeX ? This year ?
Not before texlive 2009, but probably not long after either.
-who will decide which lua modules will be available for/used with LuaTeX ? Users ? -could it be possible to install and remove any of them ? Is it the kind of thing that could be done with the TeX distribution (I'm thinking of tlmgr in TeXLive for example)
If we are assuming for the moment that luatex itself doesn't ship those libraries, that would automaticlly make it a problem for the distributions and/or individual users.
But let's get back to this later (end of summer).
Best wishes, Taco
Hi Taco, hi Hans, Winter is coming soon. What about continuing this discussion on dynamic loading lua libraries ? Jack
TeXWorld wrote:
What about continuing this discussion on dynamic loading lua libraries ?
in the current luatex beta you can load a library with 'require' we will not provide nor support libraries ourselves as the luatex engine has to be small, maintainable etc etc so it's up to yourself to make sure that your library matches the lua version used in luatex also, future releases will have some restrictions similar to running commands using write 18 so it's quite likely that for tex live 2010 there will be some guards for loading libraries (unless explicitly permitted) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hi, Hans Hagen wrote:
TeXWorld wrote:
What about continuing this discussion on dynamic loading lua libraries ?
in the current luatex beta you can load a library with 'require'
FYI, the current status is that require() handles only pure lua libraries. We hope to extend that to compiled libraries as well, but Windows is giving me a hard time: I have to alter the luatex build process rather drastically because Windows requires me to create a dll out of the embedded lua interpreter. Adding C library support on other platforms is a breeze. That new functionality will not invalidate any of Hans' comments. Best wishes, Taco
Hans Hagen a écrit :
TeXWorld wrote:
What about continuing this discussion on dynamic loading lua libraries ?
in the current luatex beta you can load a library with 'require'
ok. Currently only libraries written in pure lua work with 'require'.
we will not provide nor support libraries ourselves as the luatex engine has to be small, maintainable etc etc so it's up to yourself to make sure that your library matches the lua version used in luatex
yes of course.
also, future releases will have some restrictions similar to running commands using write 18 so it's quite likely that for tex live 2010 there will be some guards for loading libraries (unless explicitly permitted)
Hans
----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Taco Hoekwater a écrit :
Hi,
Hans Hagen wrote:
TeXWorld wrote:
What about continuing this discussion on dynamic loading lua libraries ?
in the current luatex beta you can load a library with 'require'
FYI, the current status is that require() handles only pure lua libraries. We hope to extend that to compiled libraries as well,
Fantastic !
but Windows is giving me a hard time: I have to alter the luatex build process rather drastically because Windows requires me to create a dll out of the embedded lua interpreter. Adding C library support on other platforms is a breeze.
Yes I see, I tried to use LuaXML with LuaTeX (0.44 with TexLive 2009 under Vista) and it didn't work. Do you think this dll will be ready for LuaTeX 0.50 ? Moreover, where do we have to put the files of the library ? I thnik it works when they are put in (under windows) \bin\win32 or bin\win32\lua, doesn't it ? Shouldn't they be installed somewhere else in the tds ? At last, as Hans said, there will be restrictions of course for this lua library loading. In TeX Live 2010 we could set the list of authorized external programs (it was planned for TL09 but it's not possible in fact). Do you think authorized lua libraries should be set in the same list ?
That new functionality will not invalidate any of Hans' comments.
Best wishes, Taco Thank you Taco, Thank you Hans
TeXWorld wrote:
Yes I see, I tried to use LuaXML with LuaTeX (0.44 with TexLive 2009 under Vista) and it didn't work. Do you think this dll will be ready for LuaTeX 0.50 ?
you probably need to test the latest version, not the one that ships with tex live
At last, as Hans said, there will be restrictions of course for this lua library loading. In TeX Live 2010 we could set the list of authorized external programs (it was planned for TL09 but it's not possible in fact). Do you think authorized lua libraries should be set in the same list ?
not the same list, but probably an other one as it is not a good idea to mix up functionality Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hans Hagen a écrit :
TeXWorld wrote:
Yes I see, I tried to use LuaXML with LuaTeX (0.44 with TexLive 2009 under Vista) and it didn't work. Do you think this dll will be ready for LuaTeX 0.50 ?
you probably need to test the latest version, not the one that ships with tex live
I've just updated my LuaTeX and it works now with the 0.45 version, thank you very much. Actually I have to put my lua library files in the current path. I think it also works if I put them in \bin\win32 but it doesn't (the system doesn't find the .dll file of the lua library ) if I put them in \bin\win23\lua. In fact, I would like to know where to put my lua library files, somewhere LuaTeX could find them but not in the current directory of course and not in \bin\win32 mixed up with other TeX Live binaries.
At last, as Hans said, there will be restrictions of course for this lua library loading. In TeX Live 2010 we could set the list of authorized external programs (it was planned for TL09 but it's not possible in fact). Do you think authorized lua libraries should be set in the same list ?
not the same list, but probably an other one as it is not a good idea to mix up functionality
Hans
----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Moreover, where do we have to put the files of the library ? I thnik it works when they are put in (under windows) \bin\win32 or bin\win32\lua, doesn't it ? Shouldn't they be installed somewhere else in the tds ?
Library lookups are controlled by the value of package.cpath inside the Lua interpreter (http://www.lua.org/manual/5.1/manual.html#pdf-package.cpath). It's a simple colon-separated list that defaults to a value set in luaconf.h, and can be overriden by the environment variable LUA_CPATH, so your question is really what the default value for package.cpath should be. Arthur
TeXWorld wrote:
Hans Hagen a écrit :
TeXWorld wrote:
Yes I see, I tried to use LuaXML with LuaTeX (0.44 with TexLive 2009 under Vista) and it didn't work. Do you think this dll will be ready for LuaTeX 0.50 ?
you probably need to test the latest version, not the one that ships with tex live
I've just updated my LuaTeX and it works now with the 0.45 version, thank you very much.
Earlier versions trashed package.cpath, 0.45 does not do that any more, so precompiled dlls will now work, sort of. But there is a potential problem with this: your loaded dll will use stock lua to run lua code, not the lua inside luatex. That is relevant, because the lua inside luatex has various patches (coroutines, bytecode architecture independence, an adjusted tostring) that the stock lua interpreter does not have. After some pondering (overnight) I now think that perhaps this is not an actual problem at all, but I have to do some thinking about it which is why the external dll support is not official yet.
In fact, I would like to know where to put my lua library files, somewhere LuaTeX could find them but not in the current directory of course and not in \bin\win32 mixed up with other TeX Live binaries.
For the moment: in one of the directories of package.cpath (just create a short lua script to print the value, all architectures have different defaults). In the future, there will be a way to put those dlls in the TDS structure, but that needs an extension to the callback mechanism and/or kpathsea, which is not done yet. The fact that right now you can already load external dlls under windows is more a side-effect than a feature: it works by virtue of there being a valid package.cpath, which makes the normal lua kernel kick in. There is no C support code inside luatex yet. Best wishes, Taco
Hi, Taco Hoekwater wrote:
After some pondering (overnight) I now think that perhaps this is not an actual problem at all, but I have to do some thinking about it which is why the external dll support is not official yet.
Based on this assumption, I have committed an experimental patch that enables dynamic loading for external compiled libraries unless --safer is given on the command-line. It is experimental because: * it assumes that all architectures except windows will be happy with using dlopen (windows has its own code which does not need compile- time support, but you have to have an installed lua51.dll from e.g. http://luabinaries.luaforge.net ). * I made only minimal changes to kpathsea. * there may be lua libraries out there that fail horribly. As with pure lua require() files, luatex will only use kpathsea if the library is initialized already. kpathsea changes in detail: I added the kpse_clua_format identifier that searches for files with extension .dll and .so. The texmf.cnf setting for this variable is CLUAINPUTS, and by default it has this value: CLUAINPUTS=.:$SELFAUTOLOC/lib/{$progname,$engine,}/lua// This is the best I could do on short notice: the architecture has to be in the path somewhere, and the fastest way to do that is to search below the binary directory only. One level up (lib parallel to bin) would have been nicer, but the context minimals have arch and bin swapped (texmf-linux/bin instead of bin/i386-linux) and at this early stage it would be a shame to disable the largest group of testers by default. If you know how to compile luatex from scratch, please have a go and see whether all this works for you! Best wishes, Taco
participants (4)
-
Arthur Reutenauer -
Hans Hagen -
Taco Hoekwater -
TeXWorld