April 2008 - dev-luatex - NTG Mailing Lists

Incorrect UTF-8 decoding in \luaescapestring (+ fix)
by Jonathan Sauer 08 Apr '08

08 Apr '08

Hello, the following PlainTeX example illustrates the problem: %&luatex \directlua0{% % The following character is DESERET CAPITAL LETTER LONG I, % Unicode 0x10400, encoded in UTF-8 as F0 90 90 80: local s = '\luaescapestring{𐐀}'% A % local s = '𐐀'% B for c in string.bytes(s) do texio.write_nl(c) end }% \end (A) results in: This is LuaTeX, Version snapshot-0.25.0-2008031419 (Web2C 7.5.6) (EscapeUTF8.tex! Pool contains an invalid utf-8 sequence . l.4 local s = '\luaescapestring{????} '% A ? 240 144 144 128 239 191 189 ) No pages of output. Transcript written on EscapeUTF8.log. (B) results in: This is LuaTeX, Version snapshot-0.25.0-2008031419 (Web2C 7.5.6) (EscapeUTF8.tex 240 144 144 128 ) No pages of output. Transcript written on EscapeUTF8.log. It seems that \luaescapestring does not handle long UTF-8 sequences correctly. The additional three bytes above -- 239-191-189 or EF-BF-BD -- encode Unicode character FFFD -- REPLACEMENT CHARACTER -- in UTF-8, the character LuaTeX inserts when encountering an invalid UTF-8 sequence. I think that the error lies in luatex.web, line 10911: @d unicode_incr(#)==if str_pool[#]>@"F0 then #:=#+4 else if str_pool[#]>@"E0 then #:=#+3 else if str_pool[#]>@"C0 then #:=#+2 else incr(#) Now instead of skipping the entire four-byte-sequence, only the first three bytes are skipped and the last byte, hex 80, is left for processing as the next character. Since 0x80 represents an invalid UTF-8 sequence, LuaTeX displays above error message and continues with 0xFFFD. So I think '>=' should be used, instead of '>': @d unicode_incr(#)==if str_pool[#]>=@"F0 then #:=#+4 else if str_pool[#]>=@"E0 then #:=#+3 else if str_pool[#]>=@"C0 then #:=#+2 else incr(#) I tried this modification, and the error disappeared along with the additional three bytes. Jonathan P.S: How does the bug tracker work? I tried to register some weeks ago, but never got the confirmation mail with the password. When trying to register again now, it says the username is already being used, even though non-activated accounts should be purged after a week.

2 1

Text direction in PDF mode
by Wolfgang Schuster 08 Apr '08

08 Apr '08

Hi, I tried Alephs body and textdir commands with LuaTeX and the output in DVI mode is as I want (the text is written from top to bottom and left to right). The same example in PDF mode did not work, the text is written on the left side and every new line written over the first line. %\pdfoutput=1 \hsize=8cm \vsize=8cm \bodydir RTR %\pagedir RTR \pardir RTR \textdir RTR 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 \bye Greetings, Wolfgang

2 1

another stupid question ...
by Norbert Preining 05 Apr '08

05 Apr '08

... since here are so many so intelligent people ;-) On unix I can remove the current interpreter, ie I can do #!/usr/bin/perl unlink("/usr/bin/perl"); and this will succeed (if I have permissions). On windows (again) this doesn't work. I am running the uninstall program which is a texlua script but the rmdir /q ... of bin/win32 does not succeed. The texlua script is invoked via usual win32 methods, i.e. the extension .texlua is linked to the texlua interpreter, and the .texlua extension is added to the PATHEXT to be searched, and bin/win32 is in the PATH env var. Well, in the worst case we have to live with a rogue texlua.exe somewhere. Any other ideas? Best wishes Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining <preining(a)logic.at> Vienna University of Technology Debian Developer <preining(a)debian.org> Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- EWELME (n.) The smile bestowed on you by an air hostess. --- Douglas Adams, The Meaning of Liff

5 4

writing a rmdir function in texlua
by Norbert Preining 04 Apr '08

04 Apr '08

Hi all, for the TeX Live uninstaller I am trying to write a rmdir function for win32 in texlua which should work like rm -rf on unix. My first try was: function rmdir(dirn) if os.type == 'windows' then -- we have to replace all / with \ foo = string.gsub(dirn, '/', '\\') ret = os.spawn({"rmdir", "/s", "/q", foo}) else ret = os.spawn({"rm", "-rf", dirn}) end if ret then return ret else io.stderr:write(filename..': removing '..dirn.." didn't work\n") end return ret end (No checks by now, will come later ...) But that didn't work and did spit out always the io.stderr message. Any suggestion how to do that? Or is there some code already somewhere around in the ConTeXt project? Thanks and all the best Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining <preining(a)logic.at> Vienna University of Technology Debian Developer <preining(a)debian.org> Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- Queasy but umbowed. The kind of feeling one gets when discovering a plastic compartment in a fridge in which thing are growing. --- Douglas Adams, The Meaning of Liff

4 4

Cron <luatex@metatex> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.metatex.org 02 Apr '08

02 Apr '08

U svn/trunk/src/libs/luapeg/lpeg.c U svn/trunk/src/texk/web2c/luatexdir/makecpool.c U svn/trunk/manual/luatexref-t.tex Checked out revision 1137.

1 0

Problem using umlaut in hyphenation
by Jonathan Sauer 02 Apr '08

02 Apr '08

Hello, the following PlainTeX document produces unexpected results while hyphenating the word "Streifzüge": -------------------------------------------------------------------- \hyphenation{Streif-zü-ge} Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge Streif\-zü\-ge \showhyphens{Streifzüge} \showhyphens{Streif\-zü\-ge} \bye -------------------------------------------------------------------- This is LuaTeX, Version snapshot-0.25.0-2008031419 (Web2C 7.5.6) (format=luatex 2008.3.14) 30 MAR 2008 22:29 **HyphenationProblem.tex (HyphenationProblem.tex Overfull \hbox (5.02353pt too wide) in paragraph at lines 5--9 \tenrm Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge Streifzüge| \hbox(6.94444+1.94444)x469.75499, glue set - 1.0, direction TLT .\tenrm S .\tenrm t .\tenrm r .\tenrm e .\tenrm i .etc. Underfull \hbox (badness 10000) in paragraph at lines 15--15 [][] \tenrm Streifzüge \hbox(6.94444+1.94444)x16383.99998, glue set 9793.94307, direction TLT [] Underfull \hbox (badness 10000) in paragraph at lines 17--17 [][] \tenrm Streif-zü-ge \hbox(6.94444+1.94444)x16383.99998, glue set 9793.94307, direction TLT [] [1] ) Output written on HyphenationProblem.dvi (1 page, 1052 bytes). -------------------------------------------------------------------- As you can see, I explicitly specify the hyphenation points for "Streifzüge", but for some reason LuaTeX does not hyphenate it automatically. Only when I specify the hyphenation points in the text, does the word get hyphenated. Am I doing something wrong? Jonathan

3 8

Cron <luatex@metatex> /var/www/luatex.org/www/bin/luatex-svn
by root＠www.metatex.org 01 Apr '08

01 Apr '08

U svn/trunk/src/texk/web2c/luatexdir/lua/psout.w U svn/trunk/src/texk/web2c/luatexdir/lua/mp.w U svn/trunk/src/texk/web2c/luatexdir/lua/ltexlib.c U svn/trunk/src/texk/web2c/luatexdir/lua/lmplib.c U svn/trunk/manual/luatexref-t.tex U svn/trunk/manual/luatexref-t.pdf Checked out revision 1133.

1 0