- dev-luatex - NTG Mailing Lists

Snapshot 20060908
by Taco Hoekwater 08 Sep '06

08 Sep '06

Hi again, Another friday, another snapshot! While last Wednesday's snapshot was more of a bugfix release, this one adds some extra features. * BIG CHANGE: \directlua now runs \scantokens internally. There is no longer a need to call scantokens yourself, the tex.print() output is now fed straight into a pseudo file. This is quite a bit faster as well as easier to use, but just in case you want it, the old behaviour is still present, under the new name \detokenizedlua * The texio library now has its own (partial) set of I/O commands, so that the standard lua io library is back to the distributed state (no more patched garbage collector) Note: This is really a very small change (it should not have any impact on wellbehaved input files), but I attempt to be complete in these notes, since there is no official documentation yet. * There is a new standard library available: kpse The most important function in that library is find_file: string = kpse.find_file(string filename [,string type] [,boolean mustexist]) both "type" and "mustexist" are optional, so the function has from one upto three arguments. "type" is like the '-format' argument of kpsewhich. You can run 'kpsewich -h' for a list. The default type is "tex". "mustexist" is like '-must-exist', and the default is false. If you specify "true" (or a nonzero integer), then the kpse library will search the disk as well as the ls-R databases The new library also has three other functions that map straight to kpsewhich commandline options: string = kpse.expand_path(string) string = kpse.expand_var(string) string = kpse.expand_braces(string) * There is a second callback possibility: texio.show_error_hook(string message, string|number indicator, number lineno) This command is run from inside the TeX error function, and the idea is to allow you to do some extra reporting on top of what TeX already does (none of the normal actions are removed). "message" is the formal error message tex has given to the user (the line after the "! ") "indicator" is either a filename (when it is a string) or a location indicator (a number) that can means lots of different things like a token list id or a \read number. "lineno" is the current line number This is an investigative item only, comments are welcome but do not take the syntax seriously. The final goal IMO is the total replacement of TeX's error handling routines, but that needs lots of adjustments in the web source because TeX deals with errors in a somewhat haphazard fashion (it needs unification first). The current hook is only for 'testing the water'. * It is now possible to use named items in the tex.count[] tex.dimen[] tex.toks[] subtables, so you can have tex.count.scratchcounter = 0 local enormous = tex.dimen["maxdimen"] Luatex looks up the value for you on the fly. You have to use a valid \countdef (or \dimendef, or \toksdef), anything else will generate an error. In a normal TeX file, you don't do \dimen\maxdimen, but the lua syntax seems to be a bit cleaner to me. The intention is to evntually also allow \chardefs and macros that expand into a number (todo) * The tex table has also been extended with a large list of internal parameters (partially read/write). It is possible to access about every primitive that you can use after \the *and* is a single token. So there is integer = tex.tracingcommands (\the\tracingcommands) integer = tex.prevgraf (\the\prevgraf) string = tex.output (\the\output) etc. for all of the internal parameters, but not the items that need extra arguments (like \the\scriptfont ) The subset comprising simple integer and dimension registers are writable as well as readable (stuff like \tracingcommands and \parindent). The fact that both lists are still quite limited is a result of the very different types of things that can occur after \the and also on the LHS of an assignment. The remaining primitives will be filled in when there is a real need or implementation becomes simple, whichever comes first. Have fun, Taco ------------------- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are two released files: * luatex-snapshot-20060908.tar.gz This is the source tree. * luatex-snapshot-20060908-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). This executable cannot run dynamically loaded lua dlls. Perhaps that can be fixed but I do not know how.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 08 Sep '06

08 Sep '06

U luatex/src/libs/lua51/liolib.c U luatex/src/texk/web2c/luatexdir/Makefile.in U luatex/src/texk/web2c/luatexdir/luatex.ch U luatex/src/texk/web2c/luatexdir/luastuff.c U luatex/src/texk/web2c/luatexdir/ltexiolib.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lcallbacklib.c U luatex/src/texk/web2c/luatexdir/ltexlib.c A luatex/src/texk/web2c/luatexdir/lkpselib.c U luatex/src/texk/web2c/luatexdir/luatex-api.h Checked out revision 111.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 06 Sep '06

06 Sep '06

U luatex/src/texk/web2c/luatexdir/Makefile.in U luatex/src/texk/web2c/luatexdir/luastuff.c U luatex/src/texk/web2c/luatexdir/luatex-api.h Checked out revision 108.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 06 Sep '06

06 Sep '06

D luatex/src/texk/web2c/luatexdir/luapdf.c U luatex/src/texk/web2c/luatexdir/Makefile.in A luatex/src/texk/web2c/luatexdir/lpdflib.c U luatex/src/texk/web2c/luatexdir/luastuff.c A luatex/src/texk/web2c/luatexdir/ltexiolib.c U luatex/src/texk/web2c/luatexdir/luatex.web A luatex/src/texk/web2c/luatexdir/lcallbacklib.c U luatex/src/texk/web2c/luatexdir/luatex.c A luatex/src/texk/web2c/luatexdir/ltexlib.c U luatex/src/texk/web2c/luatexdir/luatex-api.h Checked out revision 107.

1 0

Snapshot update
by Taco Hoekwater 06 Sep '06

06 Sep '06

Hi, The upload of a new snapshot will be finished in a few moments. This is a bugfix only, no new features, but I wanted to get this out of the way quickly. The list of fixed bugs/problems: * Previously, including \the inside and argument to \directlua could result in crashes and failures (due to \the changing a key variable) * texio.write() failed to clean up correctly so that the string pool became corrupted * The lua package.path contained backslashes on windows, but that doesnt work because lua interpreted them als string escapes * Assigning large values to one of the tex.toks[] registers could fill up main_memory because the memory was never cleared * tex.toks[], tex.count[] and tex.dimen[] assignments did not respect grouping * \directlua containing empty lines made the lua interpreter crash because of auto-generated \par tokens. * \string created non-utf-8 output * Bare bytes with values 0x80-0xBF in the input went unnoticed. This now raises an utf-8 encoding error * startup.lua is now searched for by the C code instead of lua 'require'. This eliminates the warning message if it cannot be found. The search path has also changed slightly: Only the current directory and the internal equivalent of 'kpsewhich --format=texmfscripts' are tried. Cheers, Taco

1 0

Snapshot 20060901
by Taco Hoekwater 06 Sep '06

06 Sep '06

Hi all, Yesterday's snapshot is in need of some documentation, so here it is. All changes are related to lua this time, and most are quite unstable / experimental. But first the things that are stabel/fixed bugs: * A VF loading bug that turned up in some of Hans' fonts has been fixed * A small series of bounds checking fixes to \ocplist has been added to prevent the system from crashing due to array indexes running out of bounds. * The Lua file searching paths are now fixed. The search path for lua script files now contains the following items (tried in order) 1. the local directory: ./?.lua (for document-specific files) 2. the items from the expansion of kpathsea's $TEXMFSCRIPTS variable, but only the parts containing 'lua' as a subpath: $TEXMFSCRIPTS<lua>/?.lua $TEXMFSCRIPTS<lua>/?/init.lua (for format-specific files) 3. the $SELFAUTOPARENT sibling directory named 'lib'. $SELFAUTOPARENT/lib/lua/5.1/?.lua $SELFAUTOPARENT/lib/lua/5.1/?/init.lua (for files that are not related to tex) The search path for dynamic libraries has only 1. the local directory: ./?.so (for document-specific files) 2. and the $SELFAUTOPARENT sibling directory named 'lib'. $SELFAUTOPARENT/lib/lua/5.1/?.so (of course the extension is .dll on windows, but .dlls do not work at the moment so it will not do you much good) * There are two functions available within a new table called texio: texio.write (luastring) texio.write_nl (luastring) both write the luastring to the same location(s) TeX writes its stuff. So if \batchmode is on, it writes only to the log, inside a \write, it prints to the current write file, etc. A read|write interface to TeX's "file selector" will follow shortly. * At startup, luatex searches for a script named startup.lua in the path list I explained above. If such a file exists, it is loaded. This happens right before the first input file needs to be opened (that is after format loading, but before any \everyjob tokens). From within the script, you can check the value of tex.formatname that is the 'format identification' as used by TEX. When the variable is equal to nil, luatex in in 'initex' mode, otherwise it will be something like: " (format=plain 2006.9.1)" Now for the experimental portion: callbacks. Here is what I have done so far: * The main reason for wanting startup.lua is file (input) re-encoding. For this purpose, it is now possible to set up a callback for luatex to execute. If you attach a Lua function to texio.input_line then from the next input line onwards, luatex will run that function whenever it needs a new input line from a text file. Your function will receive a file handle as argument, and should return either a string or nil (with nil signalling that the end of file has occurred). The trivial case is simply this: function reader (f) return f:read() end texio.input_line = reader Warning: The implementation is not totally finished yet. For the moment the file handle ("f" in the example) is a normal lua file, with a simple but important restriction: you cannot alter its value. You cannot f:close() it, or assign it a different value. luatex will eventually close the file itself. The restriction is a side-effect of a synchronisation problem with the lua garbage collector. Because of this, it also was necessary to turn off the automatic file closing code for normal lua io files (In other words: you have to close yourself all the files you opened yourself, and you should not close any files you did not open yourself). In the near future, "f" will become a special 'texio' file object and the needed functionality from the normal io library will be reimplemented. Along with that change, there will also be a callback to open (i.e. find) files, and a simple interface to the compiled-in kpathsea to use within that callback. The experimental bit is not in the i/o properties of the "reader" function: the "file handle in, line out" is very unlikely to change. It is the interfacing from luatex to the callbacks that is the experimental thing. At this moment, I have used a rather stupid system based on a name lookup: luatex searches for the "input_line" key in the "texio" table in the lua interpreter defined by the value of the register \luacallback (initially 0). If that key exists and its value is a function, then it is used, otherwise luatex sticks to the normal built-in input_ln() function. This is definately not the optimal solution, but it happens to be one of the simplest approaches possible (which is why i did it: it gives something tanglible to think about). A different (and much nicer) way would be explicit registration of callbacks, say: success = callback.register("input_line",reader) Some of the advantages of that approach: it would be a bit faster to do the actual callback; no need for the strange \luacallback register; and conflicts between various 'reader' functions would be easier to manage/spot. For future callbacks that can have multiple values (like the ocp replacements) this syntax can easily extended to something along these lines: local flist = callback.get("ocplist") -- mutate flist here -- callback.register("ocplist",flist) And the assignment could even adhere to TeX's grouping rules? But there are quite a few other ways of doing this. Any thoughts on this rather fundamental part of luatex (and especially syntax proposals) are very, very welcome. Cheers, Taco ------------------- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are two released files: * luatex-snapshot-20060901.tar.gz This is the source tree. * luatex-snapshot-20060901-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). This executable cannot run dynamically loaded lua dlls. Perhaps that can be fixed but I do not know how.

3 5

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 03 Sep '06

03 Sep '06

U luatex/src/texk/web2c/luatexdir/luastuff.c Checked out revision 104.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 01 Sep '06

01 Sep '06

U luatex/src/libs/lua51/liolib.c U luatex/src/texk/web2c/luatexdir/luastuff.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/vfpacket.c U luatex/src/texk/web2c/luatexdir/ptexlib.h Checked out revision 102.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 30 Aug '06

30 Aug '06

U luatex/src/texk/web2c/luatexdir/luatex.ch U luatex/src/texk/web2c/luatexdir/luastuff.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/luatex-api.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 98.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 28 Aug '06

28 Aug '06

U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/luatex.c U luatex/src/texk/web2c/luatexdir/luatex.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 97.

1 0