- dev-luatex - NTG Mailing Lists

beta 0.20.0 released
by Taco Hoekwater 11 Dec '07

11 Dec '07

Hi, Hi all, I have just released the new beta for luatex, with version number now at 0.20.0. This will be the last beta with big changes in it for a while: I will start work on the mplib project now, and that will be my main focus for the coming two/three months. I have planned for bug-fix releases for this beta, though. Actually, there certainly will be a 0.20.1 pretty soon, because: there are some pretty nasty bugs in the bidirectional algorithm that need fixing (sorry Idris, my brain is too blurry to solve these right now, I'll try again in a day or two); there is a node memory leak in math mode (not bad enough to be a shopstopper, but it needs fixing); the manual can be improved quite a bi (as the question from Jonathan prove!). News compared to the last beta (0.11.2) is as follows: * Completely overhauled hyphenation and ligkern application, including the addition of a new "lang" table in lua to interface to the language parameters, some extra functions in the "node" table, and a few extra callbacks ("hyphenate", "ligaturing", "kerning", "post_linebreak_filter"). There is a new chapter in the manual to document all the changes to the hyphenation and line breaking algorithms, it is simply too much to list here. * the interface of the pre_linebreak_filter,hpack_filter, vpack_filter, and pre_output_filter has changed slightly. * boxes can now get explicit attributes different from the currently active set, using a syntax like \hbox attr2=12 attr3=-1 to 12pt {Hi!} * lpeg is now at version 0.7 * a whole series of exotic bugs and compiler warnings are fixed, mostly thanks to the watchful eye of Fabrice. * texio.print now accepts multiple strings as arguments. * the lua functions os.sleep(), os.times(), os.gettimeofday() and os.tmpdir() have been added. * lua now comes with the coroutine (coco) patches from the luajit project applied. * the banner line no longer claims to be TeX. * a bunch of bugs reported on the mailing list have been fixed (I hope all of them). * (internal) we found lots of small ways to speed up lua node processing. * (internal) the node (de)allocation functions have been rewritten, so that absolutely all nodes now have a type, even the ones with variable sizes. I have uploaded source and linux/win32 binaries to http://foundry.supelec.fr/projects/luatex/ Have fun, Taco

8 28

Notes
by Jonathan Sauer 11 Dec '07

11 Dec '07

Hello, some miscellaneous notes while playing with LuaTeX 0.20.0: - Currently, the 'open_read_file' callback is only used if a 'find_read_file' callback is registered as well. Since on Apr 24 2007, Taco wrote on the mailing list (subject '\input does not look at open_read_file callback?') that this would be fixed, I suspect that this is a bug. - I want to register an open_read_file callback that is called to open the job file. \everyjob is executed after the job file has been opened, so I cannot use it to register the callback. Also, callbacks are not dumped into the format file, so I cannot register the callback while creating the format. What now? - I could use the command line switch "--lua" to execute a special script which registers the callbacks (and deal with the fact that I cannot use kpathsea to locate this script)? - I could call luatex not with the file name but like this: luatex -jobname <filename> '\input <filename>' - [insert simple and elegant solution] (On Apr 02 2007, Taco wrote on the mailing list (subject 'LaTeX and LuaTeX'), that dumping of callbacks could be added. If there was a vote on this, I would vote with 'yes' :-) - The following command line crashes LuaTeX with a bus error (yes, yes the bus that people are riding on without a ticket): luatex '\directlua0{texio.write("term and log","Hello")}' No destination or destination "term" works. The problem seems to be that LuaTeX wants to log to the log file, but the log file is not open yet. Stacktrace (not very useful, I'm afraid, since the symbols are missing): 0 libSystem.B.dylib 0x9003fd9c putc + 60 1 luatex 0x0000d3e4 0x1000 + 50148 2 luatex 0x0000d9e8 0x1000 + 51688 3 luatex 0x000a74bc 0x1000 + 681148 4 luatex 0x0015c5d0 0x1000 + 1422800 5 luatex 0x00164310 0x1000 + 1454864 6 luatex 0x0015c6a4 0x1000 + 1423012 7 luatex 0x0015bab4 0x1000 + 1419956 8 luatex 0x0015ca6c 0x1000 + 1423980 9 luatex 0x0015ab10 0x1000 + 1415952 10 luatex 0x000a5878 0x1000 + 673912 11 luatex 0x0003c850 0x1000 + 243792 12 luatex 0x000238b4 0x1000 + 141492 13 luatex 0x00023de4 0x1000 + 142820 14 luatex 0x00068004 0x1000 + 421892 15 luatex 0x0000cf04 0x1000 + 48900 16 luatex 0x00070694 0x1000 + 456340 17 luatex 0x00001d54 0x1000 + 3412 18 luatex 0x00001bfc 0x1000 + 3068 - Is it sensible to point the string functions, which are not Unicode-aware, to unicode.utf8, or are there situations (due to differences in functionality) that still require the original string functions? (I would suspect that the original functions are a bit faster) - lpeg does not seem to be UTF-8-aware. Is this impression correct? If so, how should one proceed to use it safely? I would estimate that this is mainly a problem when matching using a set ("lpeg.S") containing characters outside the first 256 code points, since when matching, a character is checked against each *byte* in the set string, not each character. Jonathan

3 2

Additions to the manual
by Jonathan Sauer 11 Dec '07

11 Dec '07

Hello, The appended .diff contains a few additions I made to the manual, revision 941 (fixed some typos, added a few notes, made one or two corrections). HTH, Jonathan

2 1

UTF-16 in \pdfoutline
by Jonathan Sauer 11 Dec '07

11 Dec '07

% The following code converts a string to UTF-16 big endian with BOM % and outputs it using \message: % We change the catcode of '%' so we can use it for modulo calculations: \begingroup \catcode`\%=12 \directlua0{\unexpanded{ function convertToUTF16(str) local result = string.char(0xFE) .. string.char(0xFF) for c in string.utfvalues(str) do if c < 0x10000 then result = result .. string.char(c / 256) .. string.char(c % 256) else c = c - 0x10000 local c1 = c / 1024 + 0xD800 local c2 = c % 1024 + 0xDC00 result = result .. string.char(c1 / 256) .. string.char(c1 % 256) .. string.char(c2 / 256) .. string.char(c2 % 256) end end tex.print('\\message{' .. result .. '}') end convertToUTF16('AäöüB!') }} \endgroup \bye This fails with 'Text line contains an invalid utf-8 sequence.' (not surprising, since the text is UTF-16 big endian). If I want to pass the UTF-16-encoded string i.e. to \pdfoutline (since PDF bookmarks can be encoded in UTF-16), how do I do this? (Maybe a callback would be useful, i.e. `convert_pdf_text') Thanks in advance, Jonathan

4 3

Hyphenation pattern information are not logged?
by Jonathan Sauer 11 Dec '07

11 Dec '07

Hello, LuaTeX does not seem to log the hyphenation memory usage anymore when in IniTeX mode. pdfTeX: ------------------------------------------------------------------------ This is pdfTeX, Version 3.141592-1.40.3 (Web2C 7.5.6) (INITEX) entering extended mode [...] Beginning to dump on file ../web2c/ktex.fmt (format=ktex 2007.12.8) 3245 strings of total length 50066 8638 memory locations dumped; current usage is 64&8395 1676 multiletter control sequences \font\nullfont=nullfont 0 words of font info for 0 preloaded fonts 14 hyphenation exceptions Hyphenation trie of length 13987 has 388 ops out of 35111 207 for language 1 181 for language 0 ------------------------------------------------------------------------ LuaTeX: ------------------------------------------------------------------------ This is LuaTeX, Version snapshot-0.20.0-2007120716 (Web2C 7.5.6) (INITEX) [...] Beginning to dump on file ../web2c/luaktex.fmt (format=luaktex 2007.12.8) 2970 strings of total length 47739 45428 memory locations dumped; current usage is 48&8787 1788 multiletter control sequences \font\nullfont=nullfont 0 preloaded fonts 0 words of active ocps \ocp\nullocp=nullocp 0 preloaded ocps \ocplist\nullocplist=[] ------------------------------------------------------------------------ I do not know if the new hyphenation mechanism still allows for logging its memory usage, and if it is still deemed useful/necessary. I just noticed this change since 0.11.2. Jonathan

2 1

Cron <luatex@www> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.bittext.nl 09 Dec '07

09 Dec '07

U svn/trunk/build.sh.cross U svn/trunk/build.sh.linux U svn/trunk/build.sh Checked out revision 951.

1 0

LuaTeX dependencies
by Joel C. Salomon 09 Dec '07

09 Dec '07

I'm considering porting LuaTεχ to a new, vaguely Uɴɪx-like, operating system. (Plan 9 from Bell Labs, if anyone cares.) What libraries/compilers/interpreters does the system have to have for this to work? E.g., I know that texexec is currently written in Ruby (which Plan 9 lacks), but Lua is straight C (which Plan 9 is built around). (If you tell me to wait a few months until things stabilize, that's fine too.) --Joel

6 8

How to insert at start of hlist/vlist?
by Heiko Oberdiek 09 Dec '07

09 Dec '07

Hello, I have some trouble to insert something at the start of a list. Example (complete example file after PS): \setbox0=\hbox{XY} \directlua0{ do local hlist = tex.box[0] local pdf_literal = node.new(8,8)% \pdfliteral pdf_literal.mode = 2 % direct pdf_literal.data = "1 0 0 rg 1 0 0 RG" local head = hlist.list % local current = hlist.list.next % works local current = hlist.list % fails hlist.list = node.insert_before(head, current, pdf_literal) end } ! LuaTeX error Attempt to node.insert_before() a non-existing node. Have I misunderstood node.insert_before()? Yours sincerely Heiko <oberdiek(a)uni-freiburg.de> PS: Complete example file that can be used with iniTeX, plain-TeX and LaTeX: \ifnum\catcode`\{=1 \else \catcode`\{=1 \catcode`\}=2 \hsize=6.5in \vsize=8.9in \parfillskip=0pt plus 1fil \fi \font\f=cmr10 \f \pdfoutput=1 \setbox0=\hbox{XY} \directlua0{ do local hlist = tex.box[0] local pdf_literal = node.new(8,8)% \pdfliteral pdf_literal.mode = 2 % direct pdf_literal.data = "1 0 0 rg 1 0 0 RG" local head = hlist.list % local current = hlist.list.next % works local current = hlist.list % fails hlist.list = node.insert_before(head, current, pdf_literal) end } \shipout\hbox{[\box0]} \csname @@end\endcsname \end

3 4

Cron <luatex@www> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.bittext.nl 07 Dec '07

07 Dec '07

U svn/trunk/src/texk/web2c/luatexdir/luatex.web U svn/trunk/src/texk/web2c/luatexdir/luatex.h U svn/trunk/src/texk/web2c/luatexdir/tex/postlinebreak.c U svn/trunk/src/texk/web2c/luatexdir/luatex.defines Checked out revision 944.

1 0

Cron <luatex@www> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.bittext.nl 07 Dec '07

07 Dec '07

U svn/trunk/manual/luatexref-t.tex U svn/trunk/manual/luatexref-t.pdf Checked out revision 941.

1 0