January 2007 - dev-luatex - NTG Mailing Lists

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 29 Jan '07

29 Jan '07

U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/luatex.ch U luatex/src/texk/web2c/luatexdir/writezip.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/lcallbacklib.c U luatex/src/texk/web2c/luatexdir/lua/lfontlib.c U luatex/src/texk/web2c/luatexdir/ptexlib.h U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/mapfile.c U luatex/src/texk/web2c/luatexdir/font/writefont.c U luatex/src/texk/web2c/luatexdir/font/texfont.c U luatex/src/texk/web2c/luatexdir/font/dofont.c U luatex/src/texk/web2c/luatexdir/font/texfont.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 265.

1 0

Snapshot 20070126
by Taco Hoekwater 27 Jan '07

27 Jan '07

Hi all, I have just released snapshot 20070126. It is yet another version with not that many changes. Most important news: * Underscores in pascal identifiers are now retained. This took quite a bit of work. The web2c conversion tools needed only small adjustments, but most of the existing C code depended on the removal of the underscores. I had to go through changing many lines. 99% of all identifiers are now written the same on the pascal side as on the C side. This makes writing interface code quite a bit less confusing. Not everything is completely clean yet, because I did not want to change the web2c-specific C files. I often took the other approach and removed the underscores from the web2c options (like 'parsefirstlinep') in the change file instead. Ugly, but predictable. There is about a dozen or so of #defines left, that map memoryword -> memory_word strpool -> str_pool etc. * The virtual font parsing and interpreting is now done in C. (this is because it needs to be interfaced with the lua-based font loader) * I am in the process of merging: the VF stuff; the TFM stuff; the lua font loading; and the pdftex font backend into one interface, so there were some smallish changes relating to that as well. Nothing worth mentioning just yet except that luatex now will read "ovf" files as well as "vf" files. * There are a few really small bugfixes for bug reports I received on this list and by email. Best, Taco ----- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are three new released files: * luatex-snapshot-20070126.tar.bz2 This is the source tree. * luatex-snapshot-20070126-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed). * luatex-snapshot-20070126-linux-i386.tar.bz2 An intel 32 linux binary (linux 2.6.17, libc 2.3.5)

2 1

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 26 Jan '07

26 Jan '07

U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/luatex.ch U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/luatex.h U luatex/src/texk/web2c/luatexdir/ptexlib.h U luatex/src/texk/web2c/luatexdir/font/writet1.c U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/vfovf.c U luatex/src/texk/web2c/luatexdir/font/texfont.c U luatex/src/texk/web2c/luatexdir/font/tfmofm.c U luatex/src/texk/web2c/luatexdir/font/dofont.c U luatex/src/texk/web2c/luatexdir/font/texfont.h U luatex/src/texk/web2c/luatexdir/luatex.defines Checked out revision 263.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 25 Jan '07

25 Jan '07

U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/Makefile.in U luatex/src/texk/web2c/luatexdir/image/epdf.c U luatex/src/texk/web2c/luatexdir/image/pdftoepdf.cc U luatex/src/texk/web2c/luatexdir/image/writepng.c U luatex/src/texk/web2c/luatexdir/image/epdf.h U luatex/src/texk/web2c/luatexdir/image/writeimg.c U luatex/src/texk/web2c/luatexdir/image/writejbig2.c U luatex/src/texk/web2c/luatexdir/image/writejpg.c U luatex/src/texk/web2c/luatexdir/textcodes.c U luatex/src/texk/web2c/luatexdir/luatex.ch U luatex/src/texk/web2c/luatexdir/writezip.c U luatex/src/texk/web2c/luatexdir/luatangle.ch U luatex/src/texk/web2c/luatexdir/luatex.mk U luatex/src/texk/web2c/luatexdir/luatexextra.in U luatex/src/texk/web2c/luatexdir/avlstuff.c U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/lcallbacklib.c U luatex/src/texk/web2c/luatexdir/lua/lstatslib.c U luatex/src/texk/web2c/luatexdir/lua/luatex.c U luatex/src/texk/web2c/luatexdir/lua/ltexlib.c U luatex/src/texk/web2c/luatexdir/lua/llualib.c U luatex/src/texk/web2c/luatexdir/lua/luainit.c U luatex/src/texk/web2c/luatexdir/lua/lfontlib.c U luatex/src/texk/web2c/luatexdir/lua/luastuff.c U luatex/src/texk/web2c/luatexdir/lua/ltexiolib.c U luatex/src/texk/web2c/luatexdir/lua/lpdflib.c U luatex/src/texk/web2c/luatexdir/ptexmac.h U luatex/src/texk/web2c/luatexdir/mathcodes.c U luatex/src/texk/web2c/luatexdir/luatex.h U luatex/src/texk/web2c/luatexdir/makecpool.c U luatex/src/texk/web2c/luatexdir/ptexlib.h U luatex/src/texk/web2c/luatexdir/luatex-api.h U luatex/src/texk/web2c/luatexdir/font/writet1.c U luatex/src/texk/web2c/luatexdir/font/writet3.c U luatex/src/texk/web2c/luatexdir/font/vfpacket.c U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/writettf.c U luatex/src/texk/web2c/luatexdir/font/mapfile.c U luatex/src/texk/web2c/luatexdir/font/writeenc.c U luatex/src/texk/web2c/luatexdir/font/subfont.c U luatex/src/texk/web2c/luatexdir/font/writefont.c A luatex/src/texk/web2c/luatexdir/font/vfovf.c U luatex/src/texk/web2c/luatexdir/font/texfont.c U luatex/src/texk/web2c/luatexdir/font/tfmofm.c U luatex/src/texk/web2c/luatexdir/font/tounicode.c U luatex/src/texk/web2c/luatexdir/font/dofont.c U luatex/src/texk/web2c/luatexdir/font/texfont.h U luatex/src/texk/web2c/luatexdir/managed-sa.c U luatex/src/texk/web2c/luatexdir/luatex.defines U luatex/src/texk/web2c/lib/texmfmp.c U luatex/src/texk/web2c/texmfmem.h U luatex/src/texk/web2c/web2c/web2c.l U luatex/src/texk/web2c/texmfmp.h Checked out revision 262.

2 1

Snapshot 20070119
by Taco Hoekwater 21 Jan '07

21 Jan '07

Hi all, I have just released snapshot 20070119. The largest new bit is the existence of a font.fonts array, that you can use to query/adjust the internal defined tex fonts. See the manual for more details. Two small new lua extensions are added as well: * os.setenv() -- to change the environment * os.exec() -- a non-returning os.execute() And a bit of cleanup in the commandline processing. Also patched is the problem with building in MacOSX that Arthur reported, and some updates to the manual. Best, Taco ----- Downloading and installation details: If you go to https://foundry.supelec.fr/frs/?group_id=10 you will see that there are two new released files: * luatex-snapshot-20070119.tar.bz2 This is the source tree. * luatex-snapshot-20070119-win32.zip A cross-compiled (mingw) windows binary. This is a web2c based binary, so it needs a texmf.cnf file (It will NOT work if you have only miktex installed).

4 4

about token processing
by Taco Hoekwater 21 Jan '07

21 Jan '07

Hi all, One of the things I have planned for the near future is an optional hook that transforms the tokens that are seen by the main control routine, and it makes sense to present that now, before I actually make changes to the program. A bit of internal documentation has to come first. Whenever TeX is ready to execute the next command, it runs the internal procedure get_x_token(). This procedure takes care of fetching the next token from the current input source (like a file or tokenlist). get_x_token() also handles all expansion, so no macros or \if statements can come through. A "token" basically consists of two integer values: a command code and a modifier (this is actually called the character code, because it most often represents a character). For characters, the command code is its category code, and the modifier the character number itself. For example, the letter "H" in a file becomes the token {cmd=11, chr=72} Control sequences are likewise converted into two parts. In this case, the character code is used to distinguish similar primitives from each other. For example, \parindent is: {cmd=79, chr=0 } and the other dimensions params only vary the chr, like \hoffset: {cmd=79, chr=18} Of course the TeX source code uses aliases for the raw numbers, the actual source code is something closer to {cmd=assign_dimen, chr=par_indent_code} (For those 'in the know': I am aware I am being a bit too informal and oversimplifying, but it is hard enough to explain already.) TeX next looks at the command code in the returned token, and jumps to a case statement with several hunderd cases. There are cases for each of the command codes, for some even different ones in each of the three processing modes: horizontal, vertical or math. (\parindent in horizontal mode behaves differently from \parindent in vertical mode) The program code inside each of these case statements take care of their own argument reading when needed, so that each command is processed by the main control function as a whole. Also, a very special exception is present: if the command code indicates a character is to be typeset while the processing mode is already horizontal, the program will jump to a special 'main_loop' case, where it will keep treating tokens as if they wre arguments of a fictional 'main_loop' command until the next command is no longer a to-be-typeset character. Only then will it jump back to the beginning of the large case statement. For example, the input \parindent 10pt Hello world \par Executes the following case statements: {vertical mode: \parindent} % the " 10pt " is read elsewhere {the letter H} % still in vertical mode {horizontal mode: the letter H} % the 'main_loop' reads "ello" {blank space } {the letter w} % the 'main_loop' reads "orld" {blank space } {\par} So much for the current state of affairs. My goal for the new luaTeX extension is twofold. One: eliminate the main_loop tricks. There is the speed-optimization trick that makes a character treat all immediately following characters as arguments, as well as the programming-logic trick that makes it read a character twice just to switch from vertical mode to horizontal mode. After these are folded back in, the case statement list would look like this: {vertical mode: \parindent} {the letter H} {the letter e} {the letter l} {the letter l} {the letter o} {blank space } {the letter w} {the letter o} {the letter r} {the letter l} {the letter d} {blank space } {\par} This has to be done with great care, because the main_loop also takes care of any otp processing and ligature building. Two: allow lua code to mutate the output of get_x_token(), before the case decision is made. Initially, the function will be called with the token from get_x_token() as argument, represented as a small lua table. The function should either return a lua table representing a to-be-processed token, or nothing at all (nil). If it returns nothing, it will be immediately called again, with yet another token from get_x_token() as argument, until it eventually does return a token. If the function does return a new token, that token will be processed in the case statement, and afterwards, the function will be called again, but now without an argument. This is repeated until it stops returning tokens. Then processing reverts back to the other branch. The point behind that roundabout calling convention is that it allows the lua function to delete, insert or buffer tokens. That in turn should make it possible to replace OTPs. Best, Taco

1 0

Re: [Dev-luatex] token processing
by Taco Hoekwater 21 Jan '07

21 Jan '07

Hi all, The text below started on a different list, then continued in private email. Since it is probably interesting for other people on this list as well, it is better to continue the discussion here. Chris Rowley wrote: > Taco > > Although I did a lot of work on the ltx3 langauge about 14 years ago I > have not seen much point in trying to `play thus with TeX' for amny > decades now. So feel free to say what you like (I know you would > anyway:-). Yes, I tend to do that :-) >>You may disagree, but I personally think the actual ltx3 syntax is >>absolutely awful. > > That it has to exist is awful: if you think it does not need to do so # > (for current TeX, please tell us ... well Frank). > The actual concrete syntax will be whatever the users want, as far as I know. > The csname naming conventon is just that, nothing to do with the semantics. As far as I can remember from the presentations, the important points are that arguments have types, macros behave like procedures, and the tail of their names function as reminder for the arguments they accept. Is that correct? Is there some documentation I should be reading? >>IMO, it requires a token processing layer > > Meaning? You mean processing other than by expansion? An example: I shiver when I see something like this (xparse.dtx): a) \def_new:NNn \xparse_already_defined_error_msg:N 1 { \xparse_error:x { Command~name~`\token_to_string:N #1'~ already~defined! } } in side TeX, you almost certainly have to do it like this, but imo it would look much better like this: b) define xparse_already_defined_error_msg (cstoken cname) { xparse_error("Command name `%N' already defined!", cname); } To go from b) to a), you could use a preprocesso. Even better would be to allow b), or a variation thereof, as input to the executable. >>on top of it just to make writing any non-trivial code bearable >>(yes, you could do that in luaTeX, using Lua, but as I am a ConTexT >>user ...) > > What does Context give you for such a token processing layer? ConTeXt does not try to force typing upon an untyped language, so it does not have the problems of ltx3. What I meant with my statement about being a ConTeXt user is that as I have no use for LaTeX code, I am not going to write a token processing layer for b) myself. Even though that could be done in luatex right now. > Yes, it is all for stuff that can be done in lua but that is not what > it was intended for. A concise definition of what it _was_ intended for would be helpful. > What you `use' (Con or La) makes no difference as a `token list programmer', > does it? Not for LaTeX2e vs. ConTeXt II (the current versions of both). But ltx3 looks like it is aiming for something completely different. Best, Taco

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 19 Jan '07

19 Jan '07

U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/lfontlib.c U luatex/src/texk/web2c/luatexdir/luatex-api.h U luatex/src/texk/web2c/luatexdir/font/luafont.c U luatex/src/texk/web2c/luatexdir/font/texfont.c U luatex/src/texk/web2c/luatexdir/font/dofont.c U luatex/src/texk/web2c/luatexdir/font/texfont.h U luatex/src/texk/web2c/luatexdir/luatex.defines U luatex/manual/luatexref-t.tex U luatex/manual/luatexref-t.pdf Checked out revision 260.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 19 Jan '07

19 Jan '07

U luatex/src/texk/web2c/luatexdir/lua/luastuff.c Checked out revision 258.

1 0

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 19 Jan '07

19 Jan '07

U luatex/build.sh.cross U luatex/src/texk/web2c/luatexdir/utils.c U luatex/src/texk/web2c/luatexdir/lua/luainit.c U luatex/src/texk/web2c/luatexdir/lua/luastuff.c U luatex/src/texk/web2c/lib/texmfmp.c Checked out revision 257.

3 3