Module searching

Heiko Oberdiek

10 Dec 2007 10 Dec '07

2:11 p.m.

Hello, in the TeX world scripts are usually installed below TDS:scripts. Lua does know a module system. Example: There is a package "foobar" with module "foobar.hello". Lua then searches in some standard places for foobar/hello.lua However LuaTeX neglects the texmf trees. Thus "package.loaders" could be extended by a kpse loader, e.g.: function kpse.module_loader(module) local file if string.find(module, ".", 1, true) then -- What to do? -- the following is just an unreliable emergency workaround: module = string.gsub(module, "^.*\%.", "") file = kpse.find_file(module .. ".lua", "texmfscripts") else file = kpse.find_file(module .. ".lua", "texmfscripts") end if file then local loader, error = loadfile(file) if loader then return loader end return "\n\t KPSE loading error:\n" .. error end return "\n\t KPSE search failed" end table.insert(package.loaders, kpse.module_loader) However, the problem are modules with dots. I haven't find a solution using kpse functions. kpsewhich --format=texmfscripts foobar/hello.lua doesn't find the file. Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function. * os.execute is not available in safer mode. * Last but not least, a kpse function is missing for searching along the modified path specification. * Portability among different operating systems. (directory specificator, ..., LUA_DIRSEP isn't available, ...) Ideas? Something like "foobar_hello.lua" isn't the best solution: * Redundancy in the full name: TDS:scripts/foobar/foobar_hello.lua * It doesn't scale well, e.g. it pollutes "package.loaded". Yours sincerely Heiko

Show replies by date

Hans Hagen

10 Dec 10 Dec

3:51 p.m.

Heiko Oberdiek wrote:

...

Hello,

in the TeX world scripts are usually installed below TDS:scripts.

Lua does know a module system.

Example: There is a package "foobar" with module "foobar.hello". Lua then searches in some standard places for foobar/hello.lua

However LuaTeX neglects the texmf trees. Thus "package.loaders" could be extended by a kpse loader, e.g.:

function kpse.module_loader(module) local file if string.find(module, ".", 1, true) then -- What to do? -- the following is just an unreliable emergency workaround: module = string.gsub(module, "^.*\%.", "") file = kpse.find_file(module .. ".lua", "texmfscripts") else file = kpse.find_file(module .. ".lua", "texmfscripts") end if file then local loader, error = loadfile(file) if loader then return loader end return "\n\t KPSE loading error:\n" .. error end return "\n\t KPSE search failed" end table.insert(package.loaders, kpse.module_loader)

However, the problem are modules with dots. I haven't find a solution using kpse functions. kpsewhich --format=texmfscripts foobar/hello.lua doesn't find the file.

well, maybe you could ask for the path variable, and loop over it there's also luainputs

...

Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function. * os.execute is not available in safer mode. * Last but not least, a kpse function is missing for searching along the modified path specification.

can't you use lua for that?

...

* Portability among different operating systems. (directory specificator, ..., LUA_DIRSEP isn't available, ...)

Ideas?

we explicitly didn't hard code these things because each user and/or macro package may want to do it different (the same for suffixes of lua files)

...

Something like "foobar_hello.lua" isn't the best solution: * Redundancy in the full name: TDS:scripts/foobar/foobar_hello.lua * It doesn't scale well, e.g. it pollutes "package.loaded".

Heiko Oberdiek

7:10 p.m.

On Mon, Dec 10, 2007 at 11:51:28PM +0100, Hans Hagen wrote:

...

Heiko Oberdiek wrote:

...
Example: There is a package "foobar" with module "foobar.hello".

...

...
I haven't found a solution using kpse functions. kpsewhich --format=texmfscripts foobar/hello.lua doesn't find the file.

well, maybe you could ask for the path variable, and loop over it

* "kpsewhich --show-path-name" doesn't help much: * It's not available in library "kpse". * It contains unwanted stuff (!!, //, ...) * kpse.expand_path (kpsewhich --expand-path) does not expand extra colons: texmf.cnf: TEXMFSCRIPTS=$TEXMF/scripts// then TEXMFSCRIPTS=.: kpsewhich --expand-path '$TEXMFSCRIPTS' only returns . * No ls-R lookup.

...

there's also luainputs

I don't see, how it could help here.

...

...
Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function. * os.execute is not available in safer mode. * Last but not least, a kpse function is missing for searching along the modified path specification.

can't you use lua for that?

Reimplementing kpse library in Lua with analyzing texmf.cnf files, path expansion, ls-R lookup, ...?

...

...
* Portability among different operating systems. (directory specificator, ..., LUA_DIRSEP isn't available, ...)

Ideas?

we explicitly didn't hard code these things because each user and/or macro package may want to do it different (the same for suffixes of lua files)

The other file types are covered: TeX files, tfm files, type 1 files, ... Missing is a proper support for lua scripts. A user/macro writer says for example: \documentclass{article} \usepackage{longtable} \usepackage[T1]{fontenc} \usepackage{lmodern} But he doesn't implement a search algorithm for finding article.cls or longtable.sty or the latin modern fonts. Also Lua has a module system, thus a user/macro writer wants to write: \directlua0{ require("foobar.hello") -- ... } without implementing, how the module is found. Lua provides four search strategies by default. These can be changed and exended, see the table "package.loader" (latest Lua docu for 5.1)

...

...
Something like "foobar_hello.lua" isn't the best solution:

Currently I am using TDS:scripts/foobar/foobar.hello.lua This doesn't follow Lua's conventions (TDS:scripts/foobar/hello.lua). But the search is easier: function kpse.module_loader(module) local script = module .. ".lua" local file = kpse.find_file(script, "texmfscripts") if file then local loader, error = loadfile(file) if loader then return loader end return "\n\tKPSE loading error:\n" .. error end return "\n\tKPSE search failed" end table.insert(package.loaders, kpse.module_loader) and avoids at least:

...

...
* It doesn't scale well, e.g. it pollutes "package.loaded".

Yours sincerely Heiko

Hans Hagen

11 Dec 11 Dec

2:25 a.m.

Heiko Oberdiek wrote:

...

Reimplementing kpse library in Lua with analyzing texmf.cnf files, path expansion, ls-R lookup, ...?

in this case i was more thinking of looking in the paths reported by kpse (given that the path variable can be expanded), so, no lookup, but looling at the file system; another option is that you just put the file alongside the tex file, lookup the tex file, replace the suffix and load the file; personally i see no need to put lua files that belong to tex in the scripts path;

...

The other file types are covered: TeX files, tfm files, type 1 files, ... Missing is a proper support for lua scripts.

lua scripts as such are like perl, ruby, whatever scripts; the texmfscript path is not for tex related (runtime embedded or whatever) scripts ... makes packaging a mess when we get a duplicate tree there, apart from clashes

...

A user/macro writer says for example: \documentclass{article} \usepackage{longtable} \usepackage[T1]{fontenc} \usepackage{lmodern} But he doesn't implement a search algorithm for finding article.cls or longtable.sty or the latin modern fonts.

Also Lua has a module system, thus a user/macro writer wants to write: \directlua0{ require("foobar.hello") -- ... } without implementing, how the module is found.

i'd say .. overload require, and put the modules in the tex tree

...

Lua provides four search strategies by default. These can be changed and exended, see the table "package.loader" (latest Lua docu for 5.1)

sure

...

...
...
Something like "foobar_hello.lua" isn't the best solution:

Currently I am using TDS:scripts/foobar/foobar.hello.lua This doesn't follow Lua's conventions (TDS:scripts/foobar/hello.lua). But the search is easier:

function kpse.module_loader(module) local script = module .. ".lua" local file = kpse.find_file(script, "texmfscripts") if file then local loader, error = loadfile(file) if loader then return loader end return "\n\tKPSE loading error:\n" .. error end return "\n\tKPSE search failed" end table.insert(package.loaders, kpse.module_loader)

and avoids at least:

...
...
* It doesn't scale well, e.g. it pollutes "package.loaded".

a solution indeed, but as said ... i wonder if texmfscripts is meant for that (not sure how easy it is for packaging too) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Heiko Oberdiek

4:03 a.m.

On Tue, Dec 11, 2007 at 10:25:57AM +0100, Hans Hagen wrote:

...

Heiko Oberdiek wrote:

...
Reimplementing kpse library in Lua with analyzing texmf.cnf files, path expansion, ls-R lookup, ...?

in this case i was more thinking of looking in the paths reported by kpse (given that the path variable can be expanded), so, no lookup, but looling at the file system; another option is that you just put the file alongside the tex file, lookup the tex file, replace the suffix and load the file; personally i see no need to put lua files that belong to tex in the scripts path;

That is not the ide behind TDS. File searching in large trees is expensive. Thus the directory structure of TDS differentiates between file types to reduce the search space.

...

...
The other file types are covered: TeX files, tfm files, type 1 files, ... Missing is a proper support for lua scripts.

lua scripts as such are like perl, ruby, whatever scripts; the texmfscript path is not for tex related (runtime embedded or whatever) scripts ... makes packaging a mess when we get a duplicate tree there, apart from clashes

A TeX distribution could precompile lua scripts and put them in a definite place, e.g. TDS:scripts/lua-compiled// (eventually adding a os component, afaik precompiled scripts aren't portable). Then texmf.cnf could contain: TEXMFSCRIPTS.lua = $TEXMF/scripts/lua-compiled//:$TEXMF/scripts// Then as first component only contains compiled lua files, few files, compared to all files below TDS:/scripts// or even TDS:tex//. Also not all Lua scripts must depend on LuaTeX. There could be standalone variants, libraries for stuff not directly related to TeX.

...

a solution indeed, but as said ... i wonder if texmfscripts is meant for that [...]

Using TDS:tex// you will have the same problem, if a module wants to have path components. At least TDS has stricter unique constraints below TDS:tex//. Two packages of the same format must not share same file names. Yours sincerely Heiko

Hans Hagen

4:26 a.m.

Heiko Oberdiek wrote:

...

That is not the ide behind TDS. File searching in large trees is expensive. Thus the directory structure of TDS differentiates between file types to reduce the search space.

sure, but tds is somewhat outdated anyway, personally i tend to flattening, for instance fonts ... i tend to use ../fonts/data/vendor/collection and put everything there -)

...

...
...
The other file types are covered: TeX files, tfm files, type 1 files, ... Missing is a proper support for lua scripts. lua scripts as such are like perl, ruby, whatever scripts; the texmfscript path is not for tex related (runtime embedded or whatever) scripts ... makes packaging a mess when we get a duplicate tree there, apart from clashes

A TeX distribution could precompile lua scripts and put them in a definite place, e.g. TDS:scripts/lua-compiled// (eventually adding a os component, afaik precompiled scripts aren't portable).

or one could use a different suffix (which is what i do, but then, i'm sure that everyone has his/her preferences)

...

Then texmf.cnf could contain: TEXMFSCRIPTS.lua = $TEXMF/scripts/lua-compiled//:$TEXMF/scripts//

no, if you want that route, the idea was/is LUAINPUTS.latex=$TEXMFSCRIPTS/latex/{lua-compiled,}// or something like that, since this is a macro package specific thing

...

Then as first component only contains compiled lua files, few files, compared to all files below TDS:/scripts// or even TDS:tex//.

Also not all Lua scripts must depend on LuaTeX. There could be standalone variants, libraries for stuff not directly related to TeX.

those can go into scripts indeed Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Taco Hoekwater

12:41 a.m.

...

...
Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function.

If you want that (or any other access to existing kpathsea stuff) just let me know, it easy to add extra bindings.

...

...
* Last but not least, a kpse function is missing for searching along the modified path specification.

That could be a bit harder, because kpathsea does not normally change variables after they have been defined, and I am not easier to actually extend the kpathsea library itself. Best wishes, Taco

Taco Hoekwater

12:49 a.m.

Taco Hoekwater wrote:

...

...
...
Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function.

If you want that (or any other access to existing kpathsea stuff) just let me know, it easy to add extra bindings.

...
...
* Last but not least, a kpse function is missing for searching along the modified path specification.

That could be a bit harder, because kpathsea does not normally change variables after they have been defined, and I am not easier to actually extend the kpathsea library itself.

Sorry, please read "not eager".

Heiko Oberdiek

3:45 a.m.

On Tue, Dec 11, 2007 at 08:41:05AM +0100, Taco Hoekwater wrote:

...

...
...
Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function.

If you want that (or any other access to existing kpathsea stuff) just let me know, it easy to add extra bindings.

"kpse.show_path" is worth a try. Then the result can be divided in separate path specifications. Under Linux I have ":" as path separator. How does --show-path behaves under other operating systems (windows, ...)? Does it use a different path separator? If yes, then a function for getting the path separator the kpathsea library uses: kpse.path_separator (either a string or a function that returns a string) The directory part of the module name is now added to each path specification. Again, is "/" the correct directory separator, also under other operating systems for kpathsea? Then the modified path specifications are merged and passed to kpse.expand_path. The result is a path list for direct script lookup. Better than kpse.expand_path would be a kpse.find_file That uses the path specification list directly for finding the script. Then the ls-R lookup speed up the process. Yours sincerely Heiko

David Kastrup

4:19 a.m.

Heiko Oberdiek writes:

...

On Tue, Dec 11, 2007 at 08:41:05AM +0100, Taco Hoekwater wrote:

...
...
...
Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function.

If you want that (or any other access to existing kpathsea stuff) just let me know, it easy to add extra bindings.

"kpse.show_path" is worth a try.

Then the result can be divided in separate path specifications.

kpse.get_path could possibly just return an already separated table. -- David Kastrup, Kriemhildstr. 15, 44793 Bochum

Heiko Oberdiek

5:46 a.m.

On Tue, Dec 11, 2007 at 12:19:45PM +0100, David Kastrup wrote:

...

Heiko Oberdiek writes:

...
"kpse.show_path" is worth a try.

Then the result can be divided in separate path specifications.

kpse.get_path could possibly just return an already separated table.

Good idea. Then kpse.find_file(string filename, table pathspecs, [boolean mustexist]) could actually find the file. Yours sincerely Heiko

6073

Age (days ago)

6074

Last active (days ago)

List overview

Download

10 comments

4 participants

participants (4)

David Kastrup
Hans Hagen
Heiko Oberdiek
Taco Hoekwater