December 2007 - dev-luatex - NTG Mailing Lists

More additions to the manual
by Jonathan Sauer 12 Dec '07

12 Dec '07

Hello, some more additions to the manual, diff against 954. Jonathan

1 0

beta 0.20.1 released
by Taco Hoekwater 12 Dec '07

12 Dec '07

Hi all, I have just uploaded the archives for luatex 0.20.1. This is a bugfix release for wednesday's beta, and it fixes the following issues: * texlua was broken on Solaris because the some C header files were loaded in the wrong order. * os.tmpdir() now accepts an argument, making it easier to use. It now generates more unique filenames when it is called repeatedly in one run. * The --luaconly commandline switch is now documented in the --help text. * The new extended hyphenation exceptions were severely broken. * \pdfprotrudechars could cause luatex to abort with an assertion failure. * There was a bug in the \span code that could cause endless loops. * The simple forms of \leaders were messed up on windows because a variable was not properly initialized. * Non-grouped \textdir commands created invalid output. * e-TeX's pseudo files were not closed properly in ini mode. * The luatex.web now weaves properly again. * I made some small improvements to the manual. I have uploaded source and linux/win32 binaries to http://foundry.supelec.fr/projects/luatex/ Have fun, Taco _______________________________________________ dev-luatex mailing list dev-luatex(a)ntg.nl http://www.ntg.nl/mailman/listinfo/dev-luatex

3 7

Cron <luatex@www> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.bittext.nl 11 Dec '07

11 Dec '07

U svn/trunk/src/libs/lua51/llimits.h U svn/trunk/src/libs/lua51/luaconf.h U svn/trunk/src/texk/web2c/luatexdir/luatex.web U svn/trunk/src/texk/web2c/luatexdir/lua/ltexlib.c U svn/trunk/src/texk/web2c/luatexdir/lua/lnodelib.c U svn/trunk/src/texk/web2c/luatexdir/tex/texnodes.c Checked out revision 958.

1 0

Module searching
by Heiko Oberdiek 11 Dec '07

11 Dec '07

Hello, in the TeX world scripts are usually installed below TDS:scripts. Lua does know a module system. Example: There is a package "foobar" with module "foobar.hello". Lua then searches in some standard places for foobar/hello.lua However LuaTeX neglects the texmf trees. Thus "package.loaders" could be extended by a kpse loader, e.g.: function kpse.module_loader(module) local file if string.find(module, ".", 1, true) then -- What to do? -- the following is just an unreliable emergency workaround: module = string.gsub(module, "^.*\%.", "") file = kpse.find_file(module .. ".lua", "texmfscripts") else file = kpse.find_file(module .. ".lua", "texmfscripts") end if file then local loader, error = loadfile(file) if loader then return loader end return "\n\t KPSE loading error:\n" .. error end return "\n\t KPSE search failed" end table.insert(package.loaders, kpse.module_loader) However, the problem are modules with dots. I haven't find a solution using kpse functions. kpsewhich --format=texmfscripts foobar/hello.lua doesn't find the file. Perhaps a search path could be hacked by using the output of kpsewhich --show-path=texmfscripts and appending the module directories to each path component. But: * LuaTeX's kpse library doesn't have a "show_path" function. * os.execute is not available in safer mode. * Last but not least, a kpse function is missing for searching along the modified path specification. * Portability among different operating systems. (directory specificator, ..., LUA_DIRSEP isn't available, ...) Ideas? Something like "foobar_hello.lua" isn't the best solution: * Redundancy in the full name: TDS:scripts/foobar/foobar_hello.lua * It doesn't scale well, e.g. it pollutes "package.loaded". Yours sincerely Heiko <oberdiek(a)uni-freiburg.de>

4 10

Cron <luatex@www> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.bittext.nl 11 Dec '07

11 Dec '07

U svn/trunk/src/texk/web2c/luatexdir/lua/ltexiolib.c U svn/trunk/manual/luatexref-t.tex U svn/trunk/manual/luatexref-t.pdf Checked out revision 955.

1 0

beta 0.20.0 released
by Taco Hoekwater 11 Dec '07

11 Dec '07

Hi, Hi all, I have just released the new beta for luatex, with version number now at 0.20.0. This will be the last beta with big changes in it for a while: I will start work on the mplib project now, and that will be my main focus for the coming two/three months. I have planned for bug-fix releases for this beta, though. Actually, there certainly will be a 0.20.1 pretty soon, because: there are some pretty nasty bugs in the bidirectional algorithm that need fixing (sorry Idris, my brain is too blurry to solve these right now, I'll try again in a day or two); there is a node memory leak in math mode (not bad enough to be a shopstopper, but it needs fixing); the manual can be improved quite a bi (as the question from Jonathan prove!). News compared to the last beta (0.11.2) is as follows: * Completely overhauled hyphenation and ligkern application, including the addition of a new "lang" table in lua to interface to the language parameters, some extra functions in the "node" table, and a few extra callbacks ("hyphenate", "ligaturing", "kerning", "post_linebreak_filter"). There is a new chapter in the manual to document all the changes to the hyphenation and line breaking algorithms, it is simply too much to list here. * the interface of the pre_linebreak_filter,hpack_filter, vpack_filter, and pre_output_filter has changed slightly. * boxes can now get explicit attributes different from the currently active set, using a syntax like \hbox attr2=12 attr3=-1 to 12pt {Hi!} * lpeg is now at version 0.7 * a whole series of exotic bugs and compiler warnings are fixed, mostly thanks to the watchful eye of Fabrice. * texio.print now accepts multiple strings as arguments. * the lua functions os.sleep(), os.times(), os.gettimeofday() and os.tmpdir() have been added. * lua now comes with the coroutine (coco) patches from the luajit project applied. * the banner line no longer claims to be TeX. * a bunch of bugs reported on the mailing list have been fixed (I hope all of them). * (internal) we found lots of small ways to speed up lua node processing. * (internal) the node (de)allocation functions have been rewritten, so that absolutely all nodes now have a type, even the ones with variable sizes. I have uploaded source and linux/win32 binaries to http://foundry.supelec.fr/projects/luatex/ Have fun, Taco

8 28

Notes
by Jonathan Sauer 11 Dec '07

11 Dec '07

Hello, some miscellaneous notes while playing with LuaTeX 0.20.0: - Currently, the 'open_read_file' callback is only used if a 'find_read_file' callback is registered as well. Since on Apr 24 2007, Taco wrote on the mailing list (subject '\input does not look at open_read_file callback?') that this would be fixed, I suspect that this is a bug. - I want to register an open_read_file callback that is called to open the job file. \everyjob is executed after the job file has been opened, so I cannot use it to register the callback. Also, callbacks are not dumped into the format file, so I cannot register the callback while creating the format. What now? - I could use the command line switch "--lua" to execute a special script which registers the callbacks (and deal with the fact that I cannot use kpathsea to locate this script)? - I could call luatex not with the file name but like this: luatex -jobname <filename> '\input <filename>' - [insert simple and elegant solution] (On Apr 02 2007, Taco wrote on the mailing list (subject 'LaTeX and LuaTeX'), that dumping of callbacks could be added. If there was a vote on this, I would vote with 'yes' :-) - The following command line crashes LuaTeX with a bus error (yes, yes the bus that people are riding on without a ticket): luatex '\directlua0{texio.write("term and log","Hello")}' No destination or destination "term" works. The problem seems to be that LuaTeX wants to log to the log file, but the log file is not open yet. Stacktrace (not very useful, I'm afraid, since the symbols are missing): 0 libSystem.B.dylib 0x9003fd9c putc + 60 1 luatex 0x0000d3e4 0x1000 + 50148 2 luatex 0x0000d9e8 0x1000 + 51688 3 luatex 0x000a74bc 0x1000 + 681148 4 luatex 0x0015c5d0 0x1000 + 1422800 5 luatex 0x00164310 0x1000 + 1454864 6 luatex 0x0015c6a4 0x1000 + 1423012 7 luatex 0x0015bab4 0x1000 + 1419956 8 luatex 0x0015ca6c 0x1000 + 1423980 9 luatex 0x0015ab10 0x1000 + 1415952 10 luatex 0x000a5878 0x1000 + 673912 11 luatex 0x0003c850 0x1000 + 243792 12 luatex 0x000238b4 0x1000 + 141492 13 luatex 0x00023de4 0x1000 + 142820 14 luatex 0x00068004 0x1000 + 421892 15 luatex 0x0000cf04 0x1000 + 48900 16 luatex 0x00070694 0x1000 + 456340 17 luatex 0x00001d54 0x1000 + 3412 18 luatex 0x00001bfc 0x1000 + 3068 - Is it sensible to point the string functions, which are not Unicode-aware, to unicode.utf8, or are there situations (due to differences in functionality) that still require the original string functions? (I would suspect that the original functions are a bit faster) - lpeg does not seem to be UTF-8-aware. Is this impression correct? If so, how should one proceed to use it safely? I would estimate that this is mainly a problem when matching using a set ("lpeg.S") containing characters outside the first 256 code points, since when matching, a character is checked against each *byte* in the set string, not each character. Jonathan

3 2

Additions to the manual
by Jonathan Sauer 11 Dec '07

11 Dec '07

Hello, The appended .diff contains a few additions I made to the manual, revision 941 (fixed some typos, added a few notes, made one or two corrections). HTH, Jonathan

2 1

UTF-16 in \pdfoutline
by Jonathan Sauer 11 Dec '07

11 Dec '07

% The following code converts a string to UTF-16 big endian with BOM % and outputs it using \message: % We change the catcode of '%' so we can use it for modulo calculations: \begingroup \catcode`\%=12 \directlua0{\unexpanded{ function convertToUTF16(str) local result = string.char(0xFE) .. string.char(0xFF) for c in string.utfvalues(str) do if c < 0x10000 then result = result .. string.char(c / 256) .. string.char(c % 256) else c = c - 0x10000 local c1 = c / 1024 + 0xD800 local c2 = c % 1024 + 0xDC00 result = result .. string.char(c1 / 256) .. string.char(c1 % 256) .. string.char(c2 / 256) .. string.char(c2 % 256) end end tex.print('\\message{' .. result .. '}') end convertToUTF16('AäöüB!') }} \endgroup \bye This fails with 'Text line contains an invalid utf-8 sequence.' (not surprising, since the text is UTF-16 big endian). If I want to pass the UTF-16-encoded string i.e. to \pdfoutline (since PDF bookmarks can be encoded in UTF-16), how do I do this? (Maybe a callback would be useful, i.e. `convert_pdf_text') Thanks in advance, Jonathan

4 3

Hyphenation pattern information are not logged?
by Jonathan Sauer 11 Dec '07

11 Dec '07

Hello, LuaTeX does not seem to log the hyphenation memory usage anymore when in IniTeX mode. pdfTeX: ------------------------------------------------------------------------ This is pdfTeX, Version 3.141592-1.40.3 (Web2C 7.5.6) (INITEX) entering extended mode [...] Beginning to dump on file ../web2c/ktex.fmt (format=ktex 2007.12.8) 3245 strings of total length 50066 8638 memory locations dumped; current usage is 64&8395 1676 multiletter control sequences \font\nullfont=nullfont 0 words of font info for 0 preloaded fonts 14 hyphenation exceptions Hyphenation trie of length 13987 has 388 ops out of 35111 207 for language 1 181 for language 0 ------------------------------------------------------------------------ LuaTeX: ------------------------------------------------------------------------ This is LuaTeX, Version snapshot-0.20.0-2007120716 (Web2C 7.5.6) (INITEX) [...] Beginning to dump on file ../web2c/luaktex.fmt (format=luaktex 2007.12.8) 2970 strings of total length 47739 45428 memory locations dumped; current usage is 48&8787 1788 multiletter control sequences \font\nullfont=nullfont 0 preloaded fonts 0 words of active ocps \ocp\nullocp=nullocp 0 preloaded ocps \ocplist\nullocplist=[] ------------------------------------------------------------------------ I do not know if the new hyphenation mechanism still allows for logging its memory usage, and if it is still deemed useful/necessary. I just noticed this change since 0.11.2. Jonathan

2 1