[Dev-luatex] Snapshot 20070202

[Taco Hoekwater]

Hi all,

This week was at least one workday too short, but I really wanted to get 
some code out, so I decided to do a snapshot anyway. I've spent quite a
bit of the past week chasing pdftex bugs, and while that is very useful,
it is also very time-consuming, so the current snapshot is (even more
than normal) rough around the edges.

* The pdftex stuff is now updated to 1.40.2, and on top of that, it has

   - an extra patch from Han The Thanh that adds a primitive to control
     the merging of fonts from external pdf images (\pdfreplacefonts),
     and a bugfix for virtual font handling.

   - an extra patch by Hartmut Henkel that removes a number of malloc()s
     and free()s from the calls to the compression library, thereby
     making luatex run a bit faster.

* luatex now happily accepts the following low-level syntax:
	
    \font\myfont = <general text> <at specifier>

   It feeds the <general text> to either kpathsea or a lua script for
   further processing after stripping the braces.

   In itself, this is not all that useful, but the reason for this
   syntax extension is that it paves the road for a macro package
   to do something like this:

    \font\myfont = {font="Times New Roman", color="\darkred",
                    encoding="ec", features="liga", size=10bp}

   without having to resort to system-specific quoting mechanisms.
   Macro (and other) expansion takes inside the braced argument.

   (I am considering adding this extension to \input as well).

* It is now possible to define a font into the font.fonts[]
   directly from lua, but becayse use is limited, I am only
   mentioning it to be complete.
	
   Syntax:	
	<number> id = font.define(<table> font)

* There is a (not very reliable yet) virtual font parser accessible
   from lua.

   Syntax:
	<table> vfdata = font.read_vf(<string> name, <number> size)

   The interpreter tries to give a reasonable representation of a
   virtual font file (the .vf file). Besides a few simple top-level
   items:

     designsize  -- in scaled points
     size        -- also in sp
     name        -- the file name
     checksum    -- the vf checksum
     header      -- the header comment  (if any)

   it also has a top-level array 'fonts' wherein each item is a mini-
   array describing one of te mapped fonts. An example makes this
   easy to understand

    "fonts" = { {"ptmr8a",655360}, {"psyr", 600000} }

   says that the first referenced font (index 1) in this virtual font
   is ptrmr8a.tfm loaded at 10pt,  and the second is psyr.tfm loaded
   at a little over 9pt. These index numbers are used by the character
   command definitions that follow.

   At top level, there is a 'characters' array, just as for a normal
   font, except that each item has only two keys:

     "width"    -- the tfm width (as stored in the .vf)
     "commands" -- an array of virtual font commands for a character.

   each of the items in the 'commands' array is itself a tiny hash.

   Each of those mini-hashes represents a single character packet
   command. Here is a 'commands' entry that contains every possible
   one at least once:

     "commands" = {
        {"push"},                     -- remember where we are
        {"right", 5000},              -- move right 0.08pt
        {"font", 1},                  -- select the fonts[1] entry
        {"setchar", 97},              -- place character 97 'a'
        {"pop"},                      -- go all the way back
        {"down", -200000},            -- move *up* about 3pt
        {"special", "pdf: 1 0 0 rg"}  -- switch to red color
        {"rule", 500000, 20000}       -- draw a bar
        {'special',"pdf: 0 g"}        -- back to black
      }


* But the big news is that you can now define your own virtual fonts
   on the fly, by adding a few statements to the table returned to
   the 'define_font' callback.

   There are a few requirements:

   - The table has to have a top-level key 'type', and it's value
     must be 'virtual'.

     The presence of this key with the specific value 'virtual'
     will trigger handling of the rest of the special virtual font
     fields in the table, but the mere existance of 'type' is
     enough to prevent luatex from looking for a virtual font
     on its own.

     Therefore, this also works 'in reverse': if you are absolutely
     certain that a font is not a virtual font, assigning the value
     'base' or 'real' to 'type' will inhibit tex from looking for a
     virtual font file, thereby saving you a disk search.

   - There must be a top-level 'fonts' array, as explained above.
     Any fonts that are not yet known to luatex at that moment,
     will be loaded automatically (Warning: do not ever use a font
     in 'fonts' with the same name you will return in the table you
     are defining right now!).

   - Each and every character in the fonts needs to have a 'commands'
     key, as explained above.

   The main point behind these restrictions is that a font is
   *either* a base font *or* a virtual font. It cannot be both
   at the same time.


Because it is not that easy to grasp these things immediately --and
from the reference documentation at that-- here is a nice little
example that is suitable for luatex --ini (adapted from Hans Hagen):

% start of virtual-demo.tex
\input plain
\pdfoutput=1
\directlua0 {
   callback.register("define_font",
     function (name,area,size)
        if name == 'cmr10-red' then
            f = font.read_tfm('cmr10',size)
            f.name = 'cmr10-red'
            f.type = 'virtual'
            f.fonts = {{'cmr10', size}}
            for i,v in pairs(f.characters) do
                if (string.char(i)):find("[tacohanshartmut]") then
                    v.commands = {
                        {'special','pdf: 1 0 0 rg'},
                        {'char',i},
                        {'special','pdf: 0 g'},
                    }
                else
                    v.commands = {
                        {'font',1},
                        {'char',i},
                    }
                end
            end
        else
          f = font.tfm_read(name,size)
        end
        return f
        end )
     }

\font\myfont = cmr10-red \myfont  This is a line of text \par
\font\myfontx= cmr10 \myfontx Here is another line of text \par

\bye
% end of virtual-demo.tex


That's it for now. Be careful. Like I said in the beginning of this
message already, there are still quite a few rough edges, and
segfaults are (unfortunately) rather likely still.

Happy TeXing,

Taco


-----

Downloading and installation details:

If you go to

      https://foundry.supelec.fr/frs/?group_id=10

you will see that there are three new released files:

* luatex-snapshot-20070202.tar.bz2
      This is the source tree.

* luatex-snapshot-20070202-win32.zip
      A cross-compiled (mingw) windows binary. This is a web2c
      based binary, so it needs a texmf.cnf file (It will NOT
      work if you have only miktex installed).

* luatex-snapshot-20070202-linux-i386.tar.bz2
      An intel 32 linux binary (linux 2.6.20-rc6, libc 2.3.5)