- dev-luatex - NTG Mailing Lists

Snapshot update
by Taco Hoekwater 06 Sep '06

06 Sep '06

Hi, The upload of a new snapshot will be finished in a few moments. This is a bugfix only, no new features, but I wanted to get this out of the way quickly. The list of fixed bugs/problems: * Previously, including \the inside and argument to \directlua could result in crashes and failures (due to \the changing a key variable) * texio.write() failed to clean up correctly so that the string pool became corrupted * The lua package.path contained backslashes on windows, but that doesnt work because lua interpreted them als string escapes * Assigning large values to one of the tex.toks[] registers could fill up main_memory because the memory was never cleared * tex.toks[], tex.count[] and tex.dimen[] assignments did not respect grouping * \directlua containing empty lines made the lua interpreter crash because of auto-generated \par tokens. * \string created non-utf-8 output * Bare bytes with values 0x80-0xBF in the input went unnoticed. This now raises an utf-8 encoding error * startup.lua is now searched for by the C code instead of lua 'require'. This eliminates the warning message if it cannot be found. The search path has also changed slightly: Only the current directory and the internal equivalent of 'kpsewhich --format=texmfscripts' are tried. Cheers, Taco

1 0

Snapshot 20060901
by Taco Hoekwater 06 Sep '06

06 Sep '06

Hi all, Yesterday's snapshot is in need of some documentation, so here it is. All changes are related to lua this time, and most are quite unstable / experimental. But first the things that are stabel/fixed bugs: * A VF loading bug that turned up in some of Hans' fonts has been fixed * A small series of bounds checking fixes to \ocplist has been added to prevent the system from crashing due to array indexes running out of bounds. * The Lua file searching paths are now fixed. The search path for lua script files now contains the following items (tried in order) 1. the local directory: ./?.lua (for document-specific files) 2. the items from the expansion of kpathsea's $TEXMFSCRIPTS variable, but only the parts containing 'lua' as a subpath: $TEXMFSCRIPTS<lua>/?.lua $TEXMFSCRIPTS<lua>/?/init.lua (for format-specific files) 3. the $SELFAUTOPARENT sibling directory named 'lib'. $SELFAUTOPARENT/lib/lua/5.1/?.lua $SELFAUTOPARENT/lib/lua/5.1/?/init.lua (for files that are not related to tex) The search path for dynamic libraries has only 1. the local directory: ./?.so (for document-specific files) 2. and the $SELFAUTOPARENT sibling directory named 'lib'. $SELFAUTOPARENT/lib/lua/5.1/?.so (of course the extension is .dll on windows, but .dlls do not work at the moment so it will not do you much good) * There are two functions available within a new table called texio: texio.write (luastring) texio.write_nl (luastring) both write the luastring to the same location(s) TeX writes its stuff. So if \batchmode is on, it writes only to the log, inside a \write, it prints to the current write file, etc. A read|write interface to TeX's "file selector" will follow shortly. * At startup, luatex searches for a script named startup.lua in the path list I explained above. If such a file exists, it is loaded. This happens right before the first input file needs to be opened (that is after format loading, but before any \everyjob tokens). From within the script, you can check the value of tex.formatname that is the 'format identification' as used by TEX. When the variable is equal to nil, luatex in in 'initex' mode, otherwise it will be something like: " (format=plain 2006.9.1)" Now for the experimental portion: callbacks. Here is what I have done so far: * The main reason for wanting startup.lua is file (input) re-encoding. For this purpose, it is now possible to set up a callback for luatex to execute. If you attach a Lua function to texio.input_line then from the next input line onwards, luatex will run that function whenever it needs a new input line from a text file. Your function will receive a file handle as argument, and should return either a string or nil (with nil signalling that the end of file has occurred). The trivial case is simply this: function reader (f) return f:read() end texio.input_line = reader Warning: The implementation is not totally finished yet. For the moment the file handle ("f" in the example) is a normal lua file, with a simple but important restriction: you cannot alter its value. You cannot f:close() it, or assign it a different value. luatex will eventually close the file itself. The restriction is a side-effect of a synchronisation problem with the lua garbage collector. Because of this, it also was necessary to turn off the automatic file closing code for normal lua io files (In other words: you have to close yourself all the files you opened yourself, and you should not close any files you did not open yourself). In the near future, "f" will become a special 'texio' file object and the needed functionality from the normal io library will be reimplemented. Along with that change, there will also be a callback to open (i.e. find) files, and a simple interface to the compiled-in kpathsea to use within that callback. The experimental bit is not in the i/o properties of the "reader" function: the "file handle in, line out" is very unlikely to change. It is the interfacing from luatex to the callbacks that is the experimental thing. At this moment, I have used a rather stupid system based on a name lookup: luatex searches for the "input_line" key in the "texio" table in the lua interpreter defined by the value of the register \luacallback (initially 0). If that key exists and its value is a function, then it is used, otherwise luatex sticks to the normal built-in input_ln() function. This is definately not the optimal solution, but it happens to be one of the simplest approaches possible (which is why i did it: it gives something tanglible to think about). A different (and much nicer) way would be explicit registration of callbacks, say: success = callback.register("input_line",reader) Some of the advantages of that approach: it would be a bit faster to do the actual callback; no need for the strange \luacallback register; and conflicts between various 'reader' functions would be easier to manage/spot. For future callbacks that can have multiple values (like the ocp replacements) this syntax can easily extended to something along these lines: local flist = callback.get("ocplist") -- mutate flist here -- callback.register("ocplist",flist) And the assignment could even adhere to TeX's grouping rules? But there are quite a few other ways of doing this. Any thoughts on this rather fundamental part of luatex (and especially syntax proposals) are very, very welcome. Cheers, Taco ------------------- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are two released files: * luatex-snapshot-20060901.tar.gz This is the source tree. * luatex-snapshot-20060901-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). This executable cannot run dynamically loaded lua dlls. Perhaps that can be fixed but I do not know how.

3 5

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 03 Sep '06

03 Sep '06

U luatex/src/texk/web2c/luatexdir/luastuff.c Checked out revision 104.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 01 Sep '06

01 Sep '06

U luatex/src/libs/lua51/liolib.c U luatex/src/texk/web2c/luatexdir/luastuff.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/vfpacket.c U luatex/src/texk/web2c/luatexdir/ptexlib.h Checked out revision 102.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 30 Aug '06

30 Aug '06

U luatex/src/texk/web2c/luatexdir/luatex.ch U luatex/src/texk/web2c/luatexdir/luastuff.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/luatex-api.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 98.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 28 Aug '06

28 Aug '06

U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/luatex.c U luatex/src/texk/web2c/luatexdir/luatex.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 97.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 25 Aug '06

25 Aug '06

U luatex/build.sh.cross U luatex/src/texk/web2c/luatexdir/Makefile.in A luatex/src/texk/web2c/luatexdir/textcodes.c U luatex/src/texk/web2c/luatexdir/luatex.web A luatex/src/texk/web2c/luatexdir/mathcodes.c U luatex/src/texk/web2c/luatexdir/luatex.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 96.

1 0

Snapshot 20060825
by Taco Hoekwater 25 Aug '06

25 Aug '06

Hi all, I have released a new snapshot just a few moments ago. The main change in this snapshot is that it uses far less memory. luatex -ini on my linux machine went from 326Mb (last week) to 93Mb (now), and "texexec luatex" now ends at 184Mb instead of over 1/2Gb. This is achieved by two big changes: * The \mathcode, \delcode, \catcode, \sfcode, \lccode and \uccode tables are now sparse arrays that are implemented in C. They are no longer part of the TeX "equivalence table" and because each had 1.1 million entries with a few memory words each, this makes a major difference. See mathcodes.c and textcodes.c if you are interested in the gory details, but be warned: the code is still extremely rough and ugly. In particular, the dump and undump routines are far from optimal, resulting in somewhat larger format files than before (approx 1.3Mb larger with unicode-letters loaded, 10k without); and the assignments do not show up when using the etex tracing routines \tracingassigns and \tracingrestores (code simply not written yet) * Single-character commands are no longer "special" internally, they are stored in the hash just like the multiletter csnames. Both changes are supposed to be transparent to the user, so if something does not work _exactly_ as before, it is a bug. There are a few memory hogs remaining (active characters and the various registers), but I will not dive into those until (much) later. Next week I will focus on the input callbacks, so that files no longer have to be utf-8 encoded. ------------------- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10&release_id=164 you will see that there are three released files: * luatex-snapshot-20060825.tar.gz This is the source tree. * luatex-snapshot-20060825-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). This executable cannot run dynamically loaded lua dlls. Perhaps that can be fixed but I do not know how. * luatex-snapshot-20060825-linux-i386.tar.gz A linux binary. This needs ld-linux.so.2, so you need a somewhat modern linux for that, but compiling from source for linux should not be a problem. The binaries should go in a 'normal' TeX binary folder like texmf/bin or texmf-win32/bin so that they can find your texmf.cnf automatically. If you put them elsewhere, you have to set the TEXMFCNF environment variable. There is no pool file, so you do not have to worry about that. For meaningful unicode input and hyphenation, you need a file that sets the \lccode etc. values for chars with a value higher than 256. One such file is part of the xetex distribution (unicode-letters.tex). Greetings, Taco

1 0

bug
by Hans Hagen 22 Aug '06

22 Aug '06

Hi Taco, As we discussed by chat ... there is a buffer overflow when the amount of chars fed into lua is large: Error: luatex.exe: buffer overflow at file ../../../../src/texk/web2c/luatexdir /utils.c, line 177 Low priority bug since it was an extreme case. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

1 0

Snapshot 20060818 released
by Taco Hoekwater 19 Aug '06

19 Aug '06

Hi all, I have made a first snapshot available, now that it processes torture.tex correctly. This is not a 'release' per se, so please do not be offended by the extreme lack of install instructions and manual. Some hints are at the end of this message. The snapshot is up to par with Aleph-RC4 (from texlive devel) and the latest pdftex beta (even newer than 20060811). However, there are some differences, to summarize: * There is no pool file (all strings are embedded) * -8bit and -translate-file are silently ignored * There is no mltex nor enctex (conflicts with aleph) * The tex functionality is always present and enabled (but see below about TeXXeT), so the asterisk or -etex switch for initex is not needed * "plus 1 filllllllll" does not generate any errors from TeX * TeXXeT (from eTeX) is not present, so the primives \TeXXeTstate \beginR \beginL \endR \endL are missing * The experimental pdftex regex match operations are removed, because that feature does not make much sense when lua is available so the primitives \pdfmatch \pdflastmatch are missing * The input translations from Aleph are not implemented, the 16 related primitives are all missing \DefaultInputMode \noDefaultInputMode \noInputMode \InputMode \DefaultOutputMode \noDefaultOutputMode \noOutputMode \OutputMode \DefaultInputTranslation \noDefaultInputTranslation \noInputTranslation \InputTranslation \DefaultOutputTranslation \noDefaultOutputTranslation \noOutputTranslation \OutputTranslation All file IO assumes UTF-8, except that all the low-ascii controls except TAB, CR en LF are written to the terminal using ^^ notation (even these are binary in the log file). * Input characters can use the full range of Unicode, but it probably only makes sense to use values above the base plane for \mathcode and \catcode assignments, since the fonts as well as the hyphenation patterns are strictly 16-bit still. * The primitives \pdfpagewidth and \pagewidth, \pdfpageheight and \pageheight, \fontcharwd and \charwd, \fontcharht and \charht, \fontchardp and \chardp, \fontcharic and \charic, are all aliases of eachother. * The new primitive \clearmarks<number> clears a marks class completely. * The experimental pdftex primitives \pdfprimitive \ifpdfprimitive \ifpdfabsnum \ifpdfabsdim and provided without the extra 'pdf' prefix, so they are simply called \primitive \ifprimitive \ifabsnum \ifabsdim instead. * I probably forgot to mention some things :-) Have fun, Taco Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10&release_id=160 you will see that there are three released files: * luatex-snapshot-20060818.tar.gz This is the source tree. * luatex-snapshot-20060818-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). This executable cannot run dynamically loaded lua dlls. Perhaps that can be fixed but I do not know how. * luatex-snapshot-20060818-linux-i386.tar.gz A linux binary. This needs ld-linux.so.2, so you need a somewhat modern linux for that, but compiling from source for linux should not be a problem. The binaries should go in a 'normal' TeX binary folder like texmf/bin or texmf-win32/bin so that they can find your texmf.cnf automatically. If you put them elsewhere, you have to set the TEXMFCNF environment variable. There is no pool file, so you do not have to worry about that. For meaningful unicode input and hyphenation, you need a file that sets the \lccode etc. values for chars with a value higher than 256. One such file is part of the xetex distribution (unicode-letters.tex). Greetings, Taco

5 9