bug with \ifcsname, ttf font spacing problems, and catcode initialisation for utf8 beyond ascii
Hello... I have been testing luatex some more and I have made the following discoveries : A) The \ifcsname macro doesn't behave as it should when you use non ascii char between \ifcsname and \endcsname, as you can see if you try to luatex this minimal file : \expandafter\def\csname A人\endcsname{success}% 1) % succeeds \csname A人\endcsname \par 2) % fails...this is a \ifcsname bug \ifcsname A人\endcsname it should write this but it doesn't \else it writes this and it shouldn't% it goes this way, sigh.... \fi \bye B) non-ascii utf8 char don't seem to have their catcodes initialized to "letter" (catcode 11). Is this normal ? Is it a bug ? a feature ? XeTeX initializes all non ascii char as letters. This is quite usefull (and quite logical too). Shouldn't it be the same in luatex, at least for the sake of compatibility ? C) I have tried using a ttf font in luatex but It was only half a success : I read the luatex manual but was really at a loss to guess how to code a function in lua to load the .ttf font, so I did a lot of googling and I finally found this luatex-wiki (started by Yannis Haralambous) : http://luatex.bluwiki.com/ I tried Yannis Haralambous's code sample to load a ttf font : \pdfoutput1 \directlua0{ callback.register('define_font', function(name, size) filename=kpse.find_file(name,"truetype fonts") if (filename) then if (size < 0) then size = (- 655.36) * size end ttffont = fontforge.to_table(fontforge.open(filename)) if ttffont then f = { } f.name = ttffont.fontname f.fullname = ttffont.names[1].names.fullname f.parameters = { } f.designsize = size f.size = size direction = 0 f.parameters.slant = 0 f.parameters.space = size * 0.25 f.parameters.space_stretch = 0.3 * size f.parameters.space_shrink = 0.1 * size f.parameters.x_height = 0.4 * size f.parameters.quad = 1.0 * size f.parameters.extra_space = 0 f.characters = { } mag = size / 2048 names_of_char = { } for char, glyph in pairs(ttffont.map.map) do names_of_char[ttffont.glyphs[glyph].name] = ttffont.map.backmap[glyph] end names_of_glyph = { } for char, glyph in pairs(ttffont.map.map) do names_of_glyph[ttffont.glyphs[glyph].name] = glyph end for char, glyph in pairs(ttffont.map.map) do glyph_table = ttffont.glyphs[glyph] f.characters[char] = { index = glyph, width = glyph_table.width * mag, name = glyph_table.name, } if glyph_table.kerns then local kerns = { } for _, kern in pairs(glyph_table.kerns) do kerns[names_of_char[kern.char]] = kern.off * mag end f.characters[char].kerns = kerns end end f.filename = filename f.type = 'real' f.format = 'truetype' f.embedding = "full" f.cidinfo = { registry = "Adobe", ordering = "Identity", supplement = 0, version = 1 } end else f = font.read_tfm(name, size) end return f end ) } \font\myfont=skullz.ttf at 8pt \myfont abcdefABCDEF %%% it prints but the all the chars are stacked upon another (they must have near 0pt width). Luatex managed to load the ttf font (it stopped whining about not finding metrics) yet, when I write something with this font, the spacing is all wrong (all the letter stack in one place). So I guess the metrics that luatex managed to produce is wrong (the char seem to have near 0 width). What could be wrong ? D) Please, could you put more code snipets in the luatex manual (this piece of code from yannis would be a good candidate) ? I welcome Yannis' decision to build a luatex wiki and it would be even nicer if someone knowledgeable wrote a little luatex macro library to do (the right way) basic things like : loading utf/opentype fonts.... To help beginners jump on the luatex bandwagon.... Best regards, Olivier Binda ps : now that the \atop bug is resolved...my 150+ page maths course does compile with luatex (yeepee). My japanese dictionnary does compile too but it doesn't produce the expected result because of the \ifcsname behavior ^_^
Olivier Binda wrote:
Hello...
I have been testing luatex some more and I have made the following discoveries :
A) The \ifcsname macro doesn't behave as it should when you use non ascii char between \ifcsname and \endcsname, as you can see if you try to luatex this minimal file :
\expandafter\def\csname A人\endcsname{success}% 1) % succeeds \csname A人\endcsname \par 2) % fails...this is a \ifcsname bug \ifcsname A人\endcsname it should write this but it doesn't \else it writes this and it shouldn't% it goes this way, sigh.... \fi \bye
indeed, this does not work; if i remember right we ran into this a while ago but it was never solved; should be on the todo list
B) non-ascii utf8 char don't seem to have their catcodes initialized to "letter" (catcode 11). Is this normal ? Is it a bug ? a feature ?
this is on purpose; you have to initialize anyway since not all characters are letters;
XeTeX initializes all non ascii char as letters. This is quite usefull (and quite logical too). Shouldn't it be the same in luatex, at least for the sake of compatibility ?
catcodes, lc/uccodes etc are to be initialized, and this is something for a macropackage to deal with
\pdfoutput1 \directlua0{ callback.register('define_font', function(name, size) filename=kpse.find_file(name,"truetype fonts") if (filename) then if (size < 0) then size = (- 655.36) * size end ttffont = fontforge.to_table(fontforge.open(filename)) if ttffont then f = { } f.name = ttffont.fontname f.fullname = ttffont.names[1].names.fullname f.parameters = { } f.designsize = size f.size = size direction = 0
^ lokks wrong to me ... this defined direction as global var
f.parameters.slant = 0 f.parameters.space = size * 0.25 f.parameters.space_stretch = 0.3 * size f.parameters.space_shrink = 0.1 * size f.parameters.x_height = 0.4 * size f.parameters.quad = 1.0 * size f.parameters.extra_space = 0 f.characters = { } mag = size / 2048
maybe this is wrong
names_of_char = { } for char, glyph in pairs(ttffont.map.map) do names_of_char[ttffont.glyphs[glyph].name] = ttffont.map.backmap[glyph] end
names_of_glyph = { } for char, glyph in pairs(ttffont.map.map) do names_of_glyph[ttffont.glyphs[glyph].name] = glyph end
for char, glyph in pairs(ttffont.map.map) do glyph_table = ttffont.glyphs[glyph]
f.characters[char] = { index = glyph, width = glyph_table.width * mag,
maybe something funny with your mag, try to set this so 10000 or so, just for testing
name = glyph_table.name, } if glyph_table.kerns then local kerns = { } for _, kern in pairs(glyph_table.kerns) do kerns[names_of_char[kern.char]] = kern.off * mag end f.characters[char].kerns = kerns end end f.filename = filename f.type = 'real' f.format = 'truetype' f.embedding = "full" f.cidinfo = { registry = "Adobe", ordering = "Identity", supplement = 0, version = 1 } end else f = font.read_tfm(name, size) end return f end ) }
\font\myfont=skullz.ttf at 8pt \myfont abcdefABCDEF %%% it prints but the all the chars are stacked upon another (they must have near 0pt width).
What could be wrong ?
a lot of things ... put some prints in the code, tracing ..
D) Please, could you put more code snipets in the luatex manual (this piece of code from yannis would be a good candidate) ?
the manual is a ref manual and will not have many examples by the time that we release the first formal version, we will provide code that works in plain tex; we will also write a book about luatex and that will hav emore samples
I welcome Yannis' decision to build a luatex wiki and it would be even nicer if someone knowledgeable wrote a little luatex macro library to do (the right way) basic things like : loading utf/opentype fonts.... To help beginners jump on the luatex bandwagon....
well, there's a truckload of experimental code in the context distribution (and i expect the context wiki also to have example code), and we will make much of that generic, but as said, mid next year is the agenda ... keep in mind that for a while font supprt changes, probably more or less stable next month eventually macro packages have to provide the basic code (we don't expect each users to write lua code for each font -) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Olivier Binda wrote:
Hello...
I have been testing luatex some more and I have made the following discoveries :
A) The \ifcsname macro doesn't behave as it should when you use non
Will get to this in the morning.
B) non-ascii utf8 char don't seem to have their catcodes initialized to "letter" (catcode 11). Is this normal ? Is it a bug ? a feature ?
This is normal. I am not convinced XeTeX is right, as some of those 'letters' are not even defined in Unicode, and some others are quite definately intended to be 'other', 'space' or even 'invalid'. So it seemed wiser to let the macro packages sort out what is needed.
C) I have tried using a ttf font in luatex but It was only half a success :
I read the luatex manual but was really at a loss to guess how to code a function in lua to load the .ttf font, so I did a lot of googling and I finally found this luatex-wiki (started by Yannis Haralambous) : http://luatex.bluwiki.com/
I tried Yannis Haralambous's code sample to load a ttf font :
IIRC, Yannis and I corresponded about an updated version a few weeks later, and I guess that never made it to the wiki.
Luatex managed to load the ttf font (it stopped whining about not finding metrics) yet, when I write something with this font, the spacing is all wrong (all the letter stack in one place). So I guess the metrics that luatex managed to produce is wrong (the char seem to have near 0 width).
What could be wrong ?
Probably the "mag" multiplier. It should be around 50.000, I guess. I'll have a closer look tomorrow.
D) Please, could you put more code snipets in the luatex manual (this piece of code from yannis would be a good candidate) ?
The reference manual will have only absolutely required examples, it is too long as it is already, and there are many more pages to come. A "normal" luatex book will be written alongside it. Best wishes, Taco
Probably the "mag" multiplier. It should be around 50.000, I guess. I'll have a closer look tomorrow.
That was is; it's fixed on the wiki. Olivier can enjoy his line of little skulls with the correct metrics ... By the way, I thought the test on size was wrong too [if (size < 0) then size = (- 655.36) * size end] because, from reading the LuaTeX reference, I assumed that a size of -1 meant an actual size of design_size, thus defaulting to 10pt == 655360sp. But after testing, it seems that it's actually not the case and that that number has to be divided by 1000. Am I mystified or should the paragraph at the end of page 50 of the manual be corrected? I do remember having read something about a factor 1000 in the past ... (like Metafont's mag parameter). Arthur
Arthur Reutenauer wrote:
Probably the "mag" multiplier. It should be around 50.000, I guess. I'll have a closer look tomorrow.
That was is; it's fixed on the wiki. Olivier can enjoy his line of little skulls with the correct metrics ...
By the way, I thought the test on size was wrong too [if (size < 0) then size = (- 655.36) * size end] because, from reading the LuaTeX reference, I assumed that a size of -1 meant an actual size of design_size, thus defaulting to 10pt == 655360sp. But after testing, it seems that it's actually not the case and that that number has to be divided by 1000. Am I mystified or should the paragraph at the end of page 50 of the manual be corrected? I do remember having read something about a factor 1000 in the past ... (like Metafont's mag parameter).
In TeX, there are two different primitive ways to load a font \font\f=cmr10 at 10pt \font\f=cmr10 scaled 1000 The default case of \font\f=cmr10 is equivalent to "scaled 1000", meaning thousand permille of the font's intended (design) size. That is what makes sure that TeX loads cmr5 at 5pt instead of 10pt. Now ,the first possibility is converted to lua as ("cmr10",655360), the second as ("cmr10", -1000). In the latter case, you have to adjust your magnifaction value according to the font's design_size (as well as for units_per_em). Because opentype font design sizes are expressed in decipoints, your calculation of 'size' should end up as if ttffont then -- otf design_size is not always set ttfont.design_size = ttfont.design_size or 100 f.designsize = 65536 * ttfont.design_size/10 if size < 0 then size = f.designsize * (-size/1000) end f.size = size I hope that this is clearer now. Best wishes, Taco
I hope that this is clearer now.
Yes, thank you very much. I somehow forgot about TeX's scaled directive. Anyway, it seems that it's only a matter of counting the powers of ten ;-)
I tried Yannis Haralambous's code sample to load a ttf font :
Redde Cæsari quæ sunt Cæsaris: the code is actually mine :-) I exchanged mail with Yannis and sent him the code, which he later edited a bit. But Hans, you must remember having seen that code too, since you looked over my shoulder when I wrote it in Münster ;-) I remember having modified it later on, especially for the mag problem (you probably have to replace the test around size with “if size < 0 then size = -size * 655360 end”, and “mag = size / 2048” with “mag = size / ttfont.units_per_em”). The setting of vertical must be an even later addition by Yannis. Anyway, I will test the code with the latest revision of LuaTeX and update the wiki as needed. Arthur
Arthur Reutenauer wrote:
I tried Yannis Haralambous's code sample to load a ttf font :
Redde Cæsari quæ sunt Cæsaris: the code is actually mine :-) I exchanged mail with Yannis and sent him the code, which he later edited a bit. But Hans, you must remember having seen that code too, since you looked over my shoulder when I wrote it in Münster ;-)
I remember having modified it later on, especially for the mag problem (you probably have to replace the test around size with “if size < 0 then size = -size * 655360 end”, and “mag = size / 2048” with “mag = size / ttfont.units_per_em”). The setting of vertical must be an even later addition by Yannis.
Anyway, I will test the code with the latest revision of LuaTeX and update the wiki as needed.
why not kick out the mag, after all it's an example and using mag complicates matters Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
participants (4)
-
Arthur Reutenauer -
Hans Hagen -
Olivier Binda -
Taco Hoekwater