- dev-luatex - NTG Mailing Lists

Snapshot 20070119
by Taco Hoekwater 21 Jan '07

21 Jan '07

Hi all, I have just released snapshot 20070119. The largest new bit is the existence of a font.fonts array, that you can use to query/adjust the internal defined tex fonts. See the manual for more details. Two small new lua extensions are added as well: * os.setenv() -- to change the environment * os.exec() -- a non-returning os.execute() And a bit of cleanup in the commandline processing. Also patched is the problem with building in MacOSX that Arthur reported, and some updates to the manual. Best, Taco ----- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are two new released files: * luatex-snapshot-20070119.tar.bz2 This is the source tree. * luatex-snapshot-20070119-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed).

4 4

about token processing
by Taco Hoekwater 21 Jan '07

21 Jan '07

Hi all, One of the things I have planned for the near future is an optional hook that transforms the tokens that are seen by the main control routine, and it makes sense to present that now, before I actually make changes to the program. A bit of internal documentation has to come first. Whenever TeX is ready to execute the next command, it runs the internal procedure get_x_token(). This procedure takes care of fetching the next token from the current input source (like a file or tokenlist). get_x_token() also handles all expansion, so no macros or \if statements can come through. A "token" basically consists of two integer values: a command code and a modifier (this is actually called the character code, because it most often represents a character). For characters, the command code is its category code, and the modifier the character number itself. For example, the letter "H" in a file becomes the token {cmd=11, chr=72} Control sequences are likewise converted into two parts. In this case, the character code is used to distinguish similar primitives from each other. For example, \parindent is: {cmd=79, chr=0 } and the other dimensions params only vary the chr, like \hoffset: {cmd=79, chr=18} Of course the TeX source code uses aliases for the raw numbers, the actual source code is something closer to {cmd=assign_dimen, chr=par_indent_code} (For those 'in the know': I am aware I am being a bit too informal and oversimplifying, but it is hard enough to explain already.) TeX next looks at the command code in the returned token, and jumps to a case statement with several hunderd cases. There are cases for each of the command codes, for some even different ones in each of the three processing modes: horizontal, vertical or math. (\parindent in horizontal mode behaves differently from \parindent in vertical mode) The program code inside each of these case statements take care of their own argument reading when needed, so that each command is processed by the main control function as a whole. Also, a very special exception is present: if the command code indicates a character is to be typeset while the processing mode is already horizontal, the program will jump to a special 'main_loop' case, where it will keep treating tokens as if they wre arguments of a fictional 'main_loop' command until the next command is no longer a to-be-typeset character. Only then will it jump back to the beginning of the large case statement. For example, the input \parindent 10pt Hello world \par Executes the following case statements: {vertical mode: \parindent} % the " 10pt " is read elsewhere {the letter H} % still in vertical mode {horizontal mode: the letter H} % the 'main_loop' reads "ello" {blank space } {the letter w} % the 'main_loop' reads "orld" {blank space } {\par} So much for the current state of affairs. My goal for the new luaTeX extension is twofold. One: eliminate the main_loop tricks. There is the speed-optimization trick that makes a character treat all immediately following characters as arguments, as well as the programming-logic trick that makes it read a character twice just to switch from vertical mode to horizontal mode. After these are folded back in, the case statement list would look like this: {vertical mode: \parindent} {the letter H} {the letter e} {the letter l} {the letter l} {the letter o} {blank space } {the letter w} {the letter o} {the letter r} {the letter l} {the letter d} {blank space } {\par} This has to be done with great care, because the main_loop also takes care of any otp processing and ligature building. Two: allow lua code to mutate the output of get_x_token(), before the case decision is made. Initially, the function will be called with the token from get_x_token() as argument, represented as a small lua table. The function should either return a lua table representing a to-be-processed token, or nothing at all (nil). If it returns nothing, it will be immediately called again, with yet another token from get_x_token() as argument, until it eventually does return a token. If the function does return a new token, that token will be processed in the case statement, and afterwards, the function will be called again, but now without an argument. This is repeated until it stops returning tokens. Then processing reverts back to the other branch. The point behind that roundabout calling convention is that it allows the lua function to delete, insert or buffer tokens. That in turn should make it possible to replace OTPs. Best, Taco

1 0

Re: [Dev-luatex] token processing
by Taco Hoekwater 21 Jan '07

21 Jan '07

Hi all, The text below started on a different list, then continued in private email. Since it is probably interesting for other people on this list as well, it is better to continue the discussion here. Chris Rowley wrote: > Taco > > Although I did a lot of work on the ltx3 langauge about 14 years ago I > have not seen much point in trying to `play thus with TeX' for amny > decades now. So feel free to say what you like (I know you would > anyway:-). Yes, I tend to do that :-) >>You may disagree, but I personally think the actual ltx3 syntax is >>absolutely awful. > > That it has to exist is awful: if you think it does not need to do so # > (for current TeX, please tell us ... well Frank). > The actual concrete syntax will be whatever the users want, as far as I know. > The csname naming conventon is just that, nothing to do with the semantics. As far as I can remember from the presentations, the important points are that arguments have types, macros behave like procedures, and the tail of their names function as reminder for the arguments they accept. Is that correct? Is there some documentation I should be reading? >>IMO, it requires a token processing layer > > Meaning? You mean processing other than by expansion? An example: I shiver when I see something like this (xparse.dtx): a) \def_new:NNn \xparse_already_defined_error_msg:N 1 { \xparse_error:x { Command~name~`\token_to_string:N #1'~ already~defined! } } in side TeX, you almost certainly have to do it like this, but imo it would look much better like this: b) define xparse_already_defined_error_msg (cstoken cname) { xparse_error("Command name `%N' already defined!", cname); } To go from b) to a), you could use a preprocesso. Even better would be to allow b), or a variation thereof, as input to the executable. >>on top of it just to make writing any non-trivial code bearable >>(yes, you could do that in luaTeX, using Lua, but as I am a ConTexT >>user ...) > > What does Context give you for such a token processing layer? ConTeXt does not try to force typing upon an untyped language, so it does not have the problems of ltx3. What I meant with my statement about being a ConTeXt user is that as I have no use for LaTeX code, I am not going to write a token processing layer for b) myself. Even though that could be done in luatex right now. > Yes, it is all for stuff that can be done in lua but that is not what > it was intended for. A concise definition of what it _was_ intended for would be helpful. > What you `use' (Con or La) makes no difference as a `token list programmer', > does it? Not for LaTeX2e vs. ConTeXt II (the current versions of both). But ltx3 looks like it is aiming for something completely different. Best, Taco

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 19 Jan '07

19 Jan '07

U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/lfontlib.c U luatex/src/texk/web2c/luatexdir/luatex-api.h U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/texfont.c U luatex/src/texk/web2c/luatexdir/font/dofont.c U luatex/src/texk/web2c/luatexdir/font/texfont.h U luatex/src/texk/web2c/luatexdir/luatex.defines U luatex/manual/luatexref-t.tex U luatex/manual/luatexref-t.pdf Checked out revision 260.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 19 Jan '07

19 Jan '07

U luatex/src/texk/web2c/luatexdir/lua/luastuff.c Checked out revision 258.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 19 Jan '07

19 Jan '07

U luatex/build.sh.cross U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/lua/luainit.c U luatex/src/texk/web2c/luatexdir/lua/luastuff.c U luatex/src/texk/web2c/lib/texmfmp.c Checked out revision 257.

3 3

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 17 Jan '07

17 Jan '07

U luatex/build.sh.linux A luatex/src/libs/luamd5 A luatex/src/libs/luamd5/md5.lua A luatex/src/libs/luamd5/md5.c A luatex/src/libs/luamd5/md5.html A luatex/src/libs/luamd5/md5tests.lua A luatex/src/libs/luamd5/md5.h A luatex/src/libs/luamd5/makefile A luatex/src/libs/luamd5/md5lib.c U luatex/src/libs/luapeg/lpeg.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/luainit.c U luatex/src/texk/web2c/luatexdir/lua/luastuff.c U luatex/src/texk/web2c/luatexdir/luatexlib.mk U luatex/src/texk/web2c/luatexdir/luatex-api.h A luatex/build.sh Checked out revision 252.

1 0

Snapshot 20070117
by Taco Hoekwater 17 Jan '07

17 Jan '07

Hi all, I have just released snapshot 20070117. A small bit of news: * the pdftex core is updated to 1.40.1, and web2c to 7.5.6 * the pdftex-specific documentation is removed * lua lpeg library updated to 0.4 ( http://www.inf.puc-rio.br/~roberto/lpeg.html ) * the C part of the lua md5 library is added to the binary ( http://www.inf.puc-rio.br/~roberto/md5/md5-5/md5.html ) * --luaonly now puts the script itself at arg[0], and preceding commandline items in the negative indices (instead of starting the count at zero). This new approach is compatible with the standalone lua interpreter. As I said earlier, it is quite possible that there are still a few bugs in the pdftex backport. There is not a whole lot of progress in this version, but now that this massive update is done, I can move on, so you can expect the next snapshot to contain some more interesting stuff. Best, Taco ----- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are two new released files: * luatex-snapshot-20070117.tar.bz2 This is the source tree. * luatex-snapshot-20070117-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed).

2 1

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 17 Jan '07

17 Jan '07

U luatex/build.sh.cross U luatex/src/texk/kpathsea/xopendir.c U luatex/src/texk/kpathsea/progname.c U luatex/src/texk/kpathsea/Makefile.in U luatex/src/texk/kpathsea/config.h U luatex/src/texk/kpathsea/tex-make.c U luatex/src/texk/kpathsea/xgetcwd.c U luatex/src/texk/web2c/lib/texmfmp.c Checked out revision 243.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 16 Jan '07

16 Jan '07

D luatex/make.cross D luatex/build.sh D luatex/make.linux D luatex/man U luatex/build.sh.cross U luatex/build.sh.linux U luatex/src/texk/web2c/cwebboot.c U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/Makefile.in U luatex/src/texk/web2c/luatexdir/image/epdf.c U luatex/src/texk/web2c/luatexdir/image/writepng.c U luatex/src/texk/web2c/luatexdir/image/writeimg.c U luatex/src/texk/web2c/luatexdir/luatangle.ch U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/luainit.c U luatex/src/texk/web2c/luatexdir/ptexmac.h U luatex/src/texk/web2c/luatexdir/luatex.h U luatex/src/texk/web2c/luatexdir/font/writet1.c U luatex/src/texk/web2c/luatexdir/font/writettf.c U luatex/src/texk/web2c/luatexdir/font/mapfile.c U luatex/src/texk/web2c/luatexdir/font/writefont.c U luatex/src/texk/web2c/luatexdir/luatex.defines U luatex/src/texk/web2c/lib/texmfmp.c U luatex/src/texk/web2c/cwebdir/comm-w2c.ch U luatex/src/texk/web2c/tangleboot.p U luatex/src/texk/web2c/texmfmp-help.h U luatex/src/texk/web2c/tangle.ch U luatex/src/texk/web2c/texmfmp.h D luatex/manual/pdftex-i.tex D luatex/manual/makefiles.cmd D luatex/manual/pdftex-t.tex D luatex/manual/samplepdf D luatex/manual/syntaxform.awk D luatex/manual/Makefile D luatex/manual/README D luatex/manual/pdftex-w.tex U luatex/manual/luatexref-t.tex U luatex/manual/luatexref-t.pdf Checked out revision 239.

1 0