- dev-luatex - NTG Mailing Lists

Utf-8 too dominant?
by David Kastrup 27 Mar '07

27 Mar '07

Hi, I just tried doing luatex -ini latex.ltx with a freshly checked out LuaTeX. The result is This is luaTeX, Version 3.141592-snapshot-2007032611 (Web2C 7.5.6) (INITEX) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/latex.ltx (/usr/local/texlive/2007/texmf-dist/tex/latex/base/texsys.cfg) ./texsys.aux found \@currdir set to: ./. Assuming \openin and \input have the same search path. Defining UNIX/DOS style filename parser. catcodes, registers, compatibility for TeX 2, parameters, LaTeX2e <2005/12/01> hacks, control, par, spacing, files, font encodings, lengths, ==================================== Local config file fonttext.cfg used ==================================== (/usr/local/texlive/2007/texmf-dist/tex/cslatex/base/fonttext.cfg (/usr/local/texlive/2007/texmf-dist/tex/latex/base/omlenc.def) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/t1enc.def) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/ot1enc.def) (/usr/local/texlive/2007/texmf-dist/tex/latex/cslatex/il2enc.def) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/omsenc.def) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/t1cmr.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/ot1cmr.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/cslatex/il2cmr.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/ot1cmss.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/ot1cmtt.fd)) ==================================== Local config file fontmath.cfg used ==================================== (/usr/local/texlive/2007/texmf-dist/tex/latex/base/fontmath.cfg (/usr/local/texlive/2007/texmf-dist/tex/latex/base/fontmath.ltx === Don't modify this file, use a .cfg file instead === (/usr/local/texlive/2007/texmf-dist/tex/latex/base/omlcmm.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/omscmsy.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/omxcmex.fd) (/usr/local/texlive/2007/texmf-dist/tex/latex/base/ucmr.fd))) ==================================== Local config file preload.cfg used ===================================== (/usr/local/texlive/2007/texmf/tex/generic/config/preload.cfg (/usr/local/texlive/2007/texmf-dist/tex/latex/base/preload.ltx)) page nos., x-ref, environments, center, verbatim, math definitions, boxes, title, sectioning, contents, floats, footnotes, index, bibliography, output, ! Buffer contains an invalid utf-8 sequence. l.7804 \lccode`\ �=`\i % dotted I ? ! Pool contains an invalid utf-8 sequence . l.7804 \lccode`\� =`\i % dotted I ? ! Buffer contains an invalid utf-8 sequence. l.7805 \uccode`\ �=`\^^9d % dotted I ? ! Pool contains an invalid utf-8 sequence . l.7805 \uccode`\� =`\^^9d % dotted I ? ! Buffer contains an invalid utf-8 sequence. l.7805 \uccode`\�=`\ � % dotted I ? ! Pool contains an invalid utf-8 sequence [...] Now the sequences in question are: \ifnum\inputlineno=\m@ne\else \lccode`\^^9d=`\i % dotted I \uccode`\^^9d=`\^^9d % dotted I \lccode`\^^9e=`\^^9e % d-bar \uccode`\^^9e=`\^^d0 % d-bar \fi In short: the buffer does not contain any illegal utf-8 sequence at all! latex.ltx consists _solely_ of ASCII characters in the range 0-127. Instead, LuaTeX barfs on "\^^9d" and similar ASCII _transliterations_ of characters which happen to be legal _characters_ in Unicode (though not legal _bytes_ in utf-8). (/usr/local/texlive/2007/texmf-dist/tex/generic/xu-hyphen/xu-bahyph.tex ! Text line contains an invalid utf-8 sequence. l.17 \lccode`\ �=0 ? ! Text line contains an invalid utf-8 sequence. l.20 \ifnum\lccode`\ �=0 % if bahyph.tex didn't change this, ? Again, the input file is purely ASCII, in this case \begingroup \expandafter\ifx\csname XeTeXrevision\endcsname\relax \else % The standard bahyph.tex is plain ASCII, so directly readable; % but we want to add patterns for n-tilde (^^f1), as generated by % bahyph.sh if the "latin1" option is given. % However, if a "latin1" version of bahyph was already present, % these would be duplicate patterns. % We'll watch the \lccode of ^^f1 so as to detect this. \lccode`\^^f1=0 \let\PATTERNS=\patterns \def\patterns{% \ifnum\lccode`\^^f1=0 % if bahyph.tex didn't change this, \lccode`\^^f1=`\^^f1 % then we can load the extra patterns here \PATTERNS{1^^f1a 1^^f1e 1^^f1o 1^^f1i 1^^f1u}% \fi \PATTERNS } \fi So we have error messages about "pool", "buffer" and "text line" containing invalid utf-8 sequences, when the input actually is just ASCII. -- David Kastrup

6 13

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 27 Mar '07

27 Mar '07

U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/ltokenlib.c Checked out revision 377.

1 0

man page for luatex
by Norbert Preining 22 Mar '07

22 Mar '07

Hi all! Not that I am a great man page writer, but I wanted to have one for Debian, so here it is, if you want to include it, improve it... Best wishes Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining <preining(a)logic.at> Università di Siena Debian Developer <preining(a)debian.org> Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- LOWESTOFT (n.) (a) The balls of wool which collect on nice new sweaters. (b) The correct name for 'navel fluff'. --- Douglas Adams, The Meaning of Liff

1 0

Wishlist item: Please set $engine ...
by Norbert Preining 22 Mar '07

22 Mar '07

Hi all! Can luatex please set the $engine variable? Current TeX Live 2007 fmtutil puts the formats into web2c/$engine/ so the luatex.fmt end up in web2c/luatex/luatex.fmt but the search path expands to: !!/var/lib/texmf/web2c/unset because $engine is unset. Can we change this? Or are there reasons *not* to do this? Best wishes Norbert ------------------------------------------------------------------------------- Dr. Norbert Preining <preining(a)logic.at> Università di Siena Debian Developer <preining(a)debian.org> Debian TeX Group gpg DSA: 0x09C5B094 fp: 14DF 2E6C 0307 BE6D AD76 A9C0 D2BF 4AA3 09C5 B094 ------------------------------------------------------------------------------- FLODIGARRY (n. Scots) An ankle-length gabardine or oilskin tarpaulin worn by deep-sea herring fishermen in Arbroath and publicans in Glasgow. --- Douglas Adams, The Meaning of Liff

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 20 Mar '07

20 Mar '07

U luatex/src/texk/web2c/luatexdir/lua/luatoken.c U luatex/src/texk/web2c/luatexdir/font/writecff.c Checked out revision 376.

1 0

Snapshot 20070309
by Taco Hoekwater 20 Mar '07

20 Mar '07

Hi all, I just released a snapshot with a fun (but not yet complete) new feature you may want to play with. What's new: * The OCP debug information is removed again. * There is a new callback that I am quite excited about myself: 'token_filter'. A bit of text from the reference manual: -------------------- This callback allows you to change the modify any lexical token that enters the \type{main_control} function before \LUATEX\ executes the associated command. Note: not all tokens can be intercepted yet, only those that are `seen' by \LUATEX's main control function. Supplemental tokens like the bodies of macro definitions and the right-hand side of register assignments are not seen. For now, this is intentional. \startfunctioncall function (table <token>) return table <token> end \stopfunctioncall Calling convention for this callback is bit more complicated then for most other callbacks. Initially, lua function will be called with the next token from \type{get_next()} as argument, represented as a small lua table. The function should either return a lua table representing a valid to-be-processed token, or something else like nil or an empty table. If your lua function does not return a table representing a valid token, it will be immediately called again with yet another token from \type{get_next()} as argument, until it eventually does return a useful token. But if the function does return a usable token, then that token will be processed by \LUATEX. Afterwards, the function will be called again, but now without an argument. This is repeated until it stops returning tokens. Then processing reverts back to the other branch. The point behind that roundabout calling convention is that it allows the lua function to buffer tokens for various uses. That in turn makes it possible to do some really advanced things like replacing OTPs. Now about that table. The table that the function will receive contains four fields: \starttabulate[|l|i|p|] \NC Key \NC type \NC Explanation \NC\FR \NC cmd \NC string \NC A representation of \LUATEX's internal command code\NC\NR \NC chr \NC number \NC The command code modifier\NC\NR \NC cs \NC string \NC If the token came from a csname, this is that csname\NC\NR \NC mod \NC character \NC A single character string representing the current processing mode. One of {\bf v}ertical, {\bf h}orizontal, display {\bf m}ath, {\bf n}o, internal {\bf V}ertical, restricted {\bf H}orizontal, or inline {\bf M}ath mode.\NC\LR \stoptabulate If you modify the table before returning it, then it is wise to return either a (\type{cmd}, \type{chr}) pair, or a \type{cs} string. That is because if both options are present, the pair has precedence and the string is ignored. On the return table, {\bf mod} is ignored always. -------------- I have no long example code yet and it is not as complete as one would want it to be because it is not intercepting everything that comes in, only the bits that are triggered by the main control loop. I decided to do a fast snapshot release for two reasons: * It is one of the easier extensions to get started with if you want to learn a bit about lua programming * I would welcome feedback on the token interface, because it will be needed in other places as well, soon. I am still not sure whether converting the command to/from strings is worth it, for example. And maybe it would be less confusing if you could return a list of tokens instead of being called repeatedly? Here is the trivial example to get you started: \directlua0{ callback.register('token_filter', function (t) return t end) } Happy TeXing, Taco ----- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are three new released files: * luatex-snapshot-20070309.tar.bz2 This is the source tree. * luatex-snapshot-20070309-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). * luatex-snapshot-20070309-linux-i386.tar.bz2 An intel 32 linux binary (2.6.17)

2 1

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 16 Mar '07

16 Mar '07

U luatex/src/libs/lua51/Makefile U luatex/src/texk/web2c/luatexdir/lua/lcallbacklib.c U luatex/src/texk/web2c/luatexdir/lua/luatoken.c U luatex/src/texk/web2c/luatexdir/lua/ltexlib.c U luatex/src/texk/web2c/luatexdir/lua/ltokenlib.c U luatex/src/texk/web2c/luatexdir/font/ttflua.c U luatex/src/texk/web2c/luatexdir/font/writecff.c U luatex/src/texk/web2c/luatexdir/font/dofont.c U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/texfont.c Checked out revision 374.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 15 Mar '07

15 Mar '07

U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/writefont.c U luatex/src/texk/web2c/luatexdir/font/vfovf.c U luatex/src/texk/web2c/luatexdir/font/texfont.c U luatex/src/texk/web2c/luatexdir/font/texfont.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 372.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 15 Mar '07

15 Mar '07

U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/Makefile.in U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/ltexlib.c A luatex/src/texk/web2c/luatexdir/lua/ltokenlib.c U luatex/src/texk/web2c/luatexdir/lua/luainit.c U luatex/src/texk/web2c/luatexdir/lua/luastuff.c U luatex/src/texk/web2c/luatexdir/font/ttflua.c U luatex/manual/luatexref-t.tex Checked out revision 368.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 14 Mar '07

14 Mar '07

U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/lcallbacklib.c U luatex/src/texk/web2c/luatexdir/lua/luatoken.c U luatex/src/texk/web2c/luatexdir/lua/ltexlib.c U luatex/src/texk/web2c/luatexdir/luatex-api.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 366.

1 0