Hi, In one of the next releases I want to start distributing some lua files under <texmf>/luatex/lua/ The intentions is that lua modules that are part of the luatex lua distribution will go there, nothing else. These are files that can be reached via the Lua require() function using the default compile-time package.path. Lua scripts have their place under <texmf>/scripts/lua already, and macro package specific lua files should remain below <texmf>/tex/, to be found via kpse.find_file(). The immediate candidates for inclusion are: from lpeg, currently not distributed at all luatex/lua/re.lua from luasocket, currently compiled into the executable: luatex/lua/socket.lua luatex/lua/ltn12.lua luatex/lua/mime.lua luatex/lua/socket/http.lua luatex/lua/socket/url.lua luatex/lua/socket/tp.lua luatex/lua/socket/ftp.lua luatex/lua/socket/smtp.lua Some others may soon arise form the luatex core itself, for example for the mapping of node (sub)type numbers to strings. These ids are static for any particular version of the executable, but they are never actually used inside the engine, so it makes sense to split them off into a lua module that can be loaded on demand. One warning and one question. First the warning, for Hans in particular: make sure you start using require() for all of the external lua modules that are embedded in luatex. As soon as this new subsystem is in place, you can no longer use functions like socket.bind() without require-ing the module first! Second the question, and that is mostly for Martin: when luatex starts shipping more stuff than just a single executable, it begins to make sense to have an installation method. It could be nice to be able $ ./configure; make; make install at toplevel, instead of $ ./build.sh ; cp <bunch of whatever to wherever> ; ... To be able to that, the current build scripts have to be rewritten into a unified Makefile.in. The new, to-be-created configure script can be rather stupid and may not even have to be autoconf-generated. The aim is to get a predictable user interface, not automating the whole big build process. For example, if "make clean" would delete the build tree (if there was one), then the "--make" commandline switch would no longer be needed. Targets like "make cross" and "make native" could replace ./build.sh.linux and ./build.sh.cross, etc. Would you (or anyone else) be willing to work on that? Best wishes, Taco
Taco Hoekwater wrote:
First the warning, for Hans in particular: make sure you start using require() for all of the external lua modules that are embedded in luatex. As soon as this new subsystem is in place, you can no longer use functions like socket.bind() without require-ing the module first!
ok, no problem for the scripts, but i have to figure out a way to embed them in the format Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hello,
First the warning, for Hans in particular: make sure you start using require() for all of the external lua modules that are embedded in luatex. As soon as this new subsystem is in place, you can no longer use functions like socket.bind() without require-ing the module first!
ok, no problem for the scripts, but i have to figure out a way to embed them in the format
I use a custom package loader and bytecode registers. A toks register is used to map Lua module names to the number of the bytecode register the module is stored in. When require()ing a module, this toks register is checked: - Module exists: The bytecode register is executed. - Modules does not exist: A new bytecode register is reserved, the module loaded using loadfile(), executed and stored in the bytecode register. Finally, the mapping is added to the toks register. That does it. I also have a macro to do <global module var> = require("<module name>") and add code to \everyjob to do this again at the start of the TeX run if called in IniTeX. Maybe this helps, Jonathan P.S: My loader is (there is a toks register \LT:T@bytecode@files which stores the mapping from module name to bytecode register, and a count register \LT:C@next@bytecode which stores the number of the next unused bytecode register; ":" and "@" are letters): local escapeChars = { ['^'] = '%^', ['$'] = '%$', ['('] = '%(', [')'] = '%)', ['%'] = '%%', ['.'] = '%.', ['['] = '%[', [']'] = '%]', ['*'] = '%*', ['+'] = '%+', ['-'] = '%-', ['?'] = '%?' } local function loadfile(file, name) local f, e = io.open(file) if not f then return f, e end local s = f:read('*a') f:close() return loadstring(s, '@' .. name) end local function loader(moduleName) local fn = moduleName .. '.klua' local t = tex.toks['LT:T@bytecode@files'] local f, n = t:match('|(' .. fn:gsub('.', escapeChars) .. ')|=|([0-9]+)|') if f then texio.write_nl('') texio.write('(', fn, ' [from register ', n, '])') return lua.bytecode[n] else rn = kpse.find_file(fn, 'tex', true) if rn then f, e = loadfile(rn, moduleName) if f then if tex.formatname == '' then local r = tex.count['LT:C@next@bytecode'] tex.count['LT:C@next@bytecode'] = r + 1 texio.write_nl('') texio.write('(', rn, ' [cached in register ', r, '])') lua.bytecode[r] = f tex.toks['LT:T@bytecode@files'] = t .. '|' .. fn .. '|=|' .. r .. '|' else texio.write_nl('') texio.write('(', rn, ')') end return f else return 'Loader error: ' .. e end else return 'Loader error: Could not find file' end end end package.loaders = { loader }
Jonathan Sauer wrote:
Hello,
First the warning, for Hans in particular: make sure you start using require() for all of the external lua modules that are embedded in luatex. As soon as this new subsystem is in place, you can no longer use functions like socket.bind() without require-ing the module first! ok, no problem for the scripts, but i have to figure out a way to embed them in the format
sorry for the confusion; in context i already put lots of stuff in bytecode registers so it's not a technical problem; more a location problem (because i want to be compatible) which in the meantime is solved by patching the mkiv file-handling lib and package.load cum suis (the main issue is actually that all the 'normal' lua modules reside in the tree and are located by the file locator, while those special libs live somewhere else and not per se known to the environment unless special env vars are set/known; fyi: i don't use kpse) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
I have a few questions about this announcement. 1st question:
Lua scripts have their place under <texmf>/scripts/lua already, I thought the lua scripts in the cont-tmf.zip distribution are residing in <texmf>/scripts/context/lua/. Is this a change or are they different files?
2nd question: In many instances the system location to place lua libraries is somewhere like /usr/local/lib/lua/5.1/lpeg.so etc. for dynamic libraries /usr/local/share/lua/5.1/re.lu etc. for lua coded modules See the discussion on page 140 of the book Programming in Lua by Roberto Ierusalimschy. Will these locations be put on the searchpath too? Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in the above, be included in the search? Finally a wish related to the build/make process, since that also seems to undergo a change. At present each compilation run also compiles ctangle. Often that program is already in the TeX distribution. Is it possible to use that instead and resort to compilation only when it cannot be found? Hans van der Meer On 4 jan 2009, at 15:10, Taco Hoekwater wrote:
Hi,
In one of the next releases I want to start distributing some lua files under <texmf>/luatex/lua/
The intentions is that lua modules that are part of the luatex lua distribution will go there, nothing else. These are files that can be reached via the Lua require() function using the default compile-time package.path. Lua scripts have their place under <texmf>/scripts/lua already, and macro package specific lua files should remain below <texmf>/tex/, to be found via kpse.find_file().
The immediate candidates for inclusion are:
from lpeg, currently not distributed at all
luatex/lua/re.lua
from luasocket, currently compiled into the executable:
luatex/lua/socket.lua luatex/lua/ltn12.lua luatex/lua/mime.lua luatex/lua/socket/http.lua luatex/lua/socket/url.lua luatex/lua/socket/tp.lua luatex/lua/socket/ftp.lua luatex/lua/socket/smtp.lua
Some others may soon arise form the luatex core itself, for example for the mapping of node (sub)type numbers to strings. These ids are static for any particular version of the executable, but they are never actually used inside the engine, so it makes sense to split them off into a lua module that can be loaded on demand.
One warning and one question.
First the warning, for Hans in particular: make sure you start using require() for all of the external lua modules that are embedded in luatex. As soon as this new subsystem is in place, you can no longer use functions like socket.bind() without require-ing the module first!
Second the question, and that is mostly for Martin: when luatex starts shipping more stuff than just a single executable, it begins to make sense to have an installation method. It could be nice to be able
$ ./configure; make; make install
at toplevel, instead of
$ ./build.sh ; cp <bunch of whatever to wherever> ; ...
To be able to that, the current build scripts have to be rewritten into a unified Makefile.in. The new, to-be-created configure script can be rather stupid and may not even have to be autoconf-generated.
The aim is to get a predictable user interface, not automating the whole big build process. For example, if "make clean" would delete the build tree (if there was one), then the "--make" commandline switch would no longer be needed. Targets like "make cross" and "make native" could replace ./build.sh.linux and ./build.sh.cross, etc.
Would you (or anyone else) be willing to work on that?
Best wishes, Taco
_______________________________________________ dev-luatex mailing list dev-luatex@ntg.nl http://www.ntg.nl/mailman/listinfo/dev-luatex
I thought the lua scripts in the cont-tmf.zip distribution are residing in <texmf>/scripts/context/lua/. Is this a change or are they different files?
Different files, see Taco's message from 17:31.
Will these locations be put on the searchpath too? Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in the above, be included in the search?
I don't know how the developers will solve that, but it's obvious that it needs to be handled somehow (maybe through kpathsea? Yet another variable name in texmf.cnf ...). Arthur
Arthur Reutenauer wrote:
I thought the lua scripts in the cont-tmf.zip distribution are residing in <texmf>/scripts/context/lua/. Is this a change or are they different files?
Different files, see Taco's message from 17:31.
Will these locations be put on the searchpath too? Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in the above, be included in the search?
I don't know how the developers will solve that, but it's obvious that it needs to be handled somehow (maybe through kpathsea? Yet another variable name in texmf.cnf ...).
Kpathsea has no trouble finding lua files in the tex/ path, but I remember talking to Karl about a specific variable a while back. That would not be a big deal, there are so many of them already that one extra hardly matters. The issue about <texmf>/luatex/lua is that luatex should be able to find these files even if kpathsea initialization is off or delayed. That is why they have to become part of the 'compiled-in' package.path. If|Once kpathsea is active, luatex could simply use it for finding the lua modules, but that may never happen and I want to have a strong guarantee about the discoverability of the files in <texmf>/luatex/lua. Best wishes, Taco
Hans van der Meer wrote:
I have a few questions about this announcement.
1st question:
Lua scripts have their place under <texmf>/scripts/lua already, I thought the lua scripts in the cont-tmf.zip distribution are residing in <texmf>/scripts/context/lua/. Is this a change or are they different files?
only some mtx-* scripts; they use libs that are in tex/context/base and that will not change; macro packages should put their code under their designated paths (for latex there is also a subpath for authors, for context there is the /third etc subpaths for users)
2nd question: In many instances the system location to place lua libraries is somewhere like /usr/local/lib/lua/5.1/lpeg.so etc. for dynamic libraries /usr/local/share/lua/5.1/re.lu etc. for lua coded modules See the discussion on page 140 of the book Programming in Lua by Roberto Ierusalimschy. Will these locations be put on the searchpath too? Or will the environment variables LUA_PATH and LUA_CPATH, mentioned in the above, be included in the search?
loading of extra libraries (the core libs will remain statically linked) will be looked into later Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
On So, 04 Jan 2009, Taco Hoekwater wrote:
lua files under <texmf>/luatex/lua/
The intentions is that lua modules that are part of the luatex lua distribution will go there, nothing else. These are files that can be reached via the Lua require() function using the default compile-time package.path. Lua scripts have their place under <texmf>/scripts/lua already, and macro package specific lua files should remain below <texmf>/tex/, to be found via kpse.find_file().
Hmmm, what about <texmf>/scripts/lua/luatex/ ? I consider that a better candidate then creating another toplevel hierarchy. Is there a particular need for that? The above suggestion would serve the same purpose, and anyone can anyway install into <texmf>/luatex/lua, too, if one wants to. So I don't see any direct advantage.
Second the question, and that is mostly for Martin: when luatex starts shipping more stuff than just a single executable, it begins to make sense to have an installation method. It could be nice to be able
$ ./configure; make; make install
at toplevel, instead of
$ ./build.sh ; cp <bunch of whatever to wherever> ; ...
OHHHH PLEASE!!!!!! YES!!!!!
To be able to that, the current build scripts have to be rewritten into a unified Makefile.in. The new, to-be-created configure script can be rather stupid and may not even have to be autoconf-generated.
If you do the step please make sure that dropping it into texlive trees
makes it easy already. Maybe that is already planned/done/whatever,
wanted only to remind it ;-)
Best wishes
Norbert
-------------------------------------------------------------------------------
Dr. Norbert Preining
Norbert Preining wrote:
Hmmm, what about <texmf>/scripts/lua/luatex/ ? I consider that a better candidate then creating another toplevel hierarchy.
it also depends on the variables set; what happens if one has files with the same names in scripts/lua/luatex and scripts/lua/whatever i think that the idea was to have that toplevel path for libs that belong to luatex itself, and not to some package Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hi Norbert, Norbert Preining wrote:
Hmmm, what about <texmf>/scripts/lua/luatex/ ? I consider that a better candidate then creating another toplevel hierarchy.
Nothing has been done yet, but I favour <texmf>/luatex, and Karl agreed with me :-) The files in question are not scripts. For example, take socket.lua. This is the lua part of the socket implementation, of which the C side is compiled into the executable (as socket.a). socket.lua is not a standalone package, and it has to have the same version exactly as the embedded socket.a, both are distributed by luatex itself, and both come from upstream. And there is a precedent: bibtex, dvips, omega, all have similar top-level directories for runtime files that do not depend on macro packages. Best wishes, Taco
On So, 04 Jan 2009, Taco Hoekwater wrote:
Nothing has been done yet, but I favour <texmf>/luatex, and Karl agreed with me :-)
Ok, that is a good indicator ;-)
The files in question are not scripts. For example, take socket.lua. This is the lua part of the socket implementation, of which the C side is compiled into the executable (as socket.a). socket.lua is not a standalone package, and it has to have the same version exactly as the embedded socket.a, both are distributed by luatex itself, and both come from upstream.
Ok, I see the difference now. Fine with me.
And there is a precedent: bibtex, dvips, omega, all have similar top-level directories for runtime files that do not depend on macro packages.
Agreed.
Sorry for the noise.
Best wishes
Norbert
-------------------------------------------------------------------------------
Dr. Norbert Preining
participants (6)
-
Arthur Reutenauer -
Hans Hagen -
Hans van der Meer -
Jonathan Sauer -
Norbert Preining -
Taco Hoekwater