- dev-luatex - NTG Mailing Lists

Cron <luatex@server> /home/luatex/luatex-svn
by root＠aanhet.net 06 Apr '07

06 Apr '07

U luatex/src/texk/web2c/luatexdir/Makefile.in U luatex/src/texk/web2c/luatexdir/luatex.web U luatex/src/texk/web2c/luatexdir/lua/lcallbacklib.c A luatex/src/texk/web2c/luatexdir/lua/luawhatsits.c A luatex/src/texk/web2c/luatexdir/lua/nodes.h U luatex/src/texk/web2c/luatexdir/lua/luatoken.c U luatex/src/texk/web2c/luatexdir/lua/ltokenlib.c A luatex/src/texk/web2c/luatexdir/lua/luanode.c U luatex/src/texk/web2c/luatexdir/font/ttflua.c U luatex/src/texk/web2c/luatexdir/font/ttfotf.c U luatex/src/texk/web2c/luatexdir/font/splinefont.h U luatex/src/texk/web2c/luatexdir/luatex.defines U luatex/manual/luatexref-t.tex Checked out revision 386.

1 0

Minimal example of using OTF/TTF?
by Geoffrey Alan Washburn 06 Apr '07

06 Apr '07

There is probably a really simple answer to this, but what is the current method for loading OpenType/TrueType fonts in luatex? I've tried a few things that do not appear to work, and I haven't been able to find any complete examples of how to do so. \pdfoutput=1 \font\myfont={Vera.ttf} % This doesn't work %\font\myfont={cmr10} % This does work {\myfont Testing} \end % luatex --fmt=plain foo.tex This is luaTeX, Version 3.141592-snapshot-2007040613 (Web2C 7.5.6) (foo.tex ! Font \myfont=Vera.ttf not loadable: Metric (TFM/OFM) file not found or bad. <to be read again> \par l.4 Where Vera.ttf is in current directory. My impression from the manual (page 38) was that luatex could construct metrics directly from the font without needing something like otftotfm. I also tried registering a define_font callback that called the font.read_ttf function, but that gave similar results. I'm using luatex build from the subversion repository (currently at revision 386). -- [Geoff Washburn|geoffw@cis.upenn.edu|http://www.cis.upenn.edu/~geoffw/]

2 1

Benchmark.
by David Kastrup 06 Apr '07

06 Apr '07

Hi, latex -ini latex.ltx gives real 0m0.498s user 0m0.080s sys 0m0.408s luatex -ini latex.8bit (where the latter just sets up the bytes->utf8 conversion through \directlua0{ callback.register("process_input_buffer", function(buf) return unicode.utf8.char(unicode.latin1.byte(buf,1,-1)) end)} like in the last mail and then loads LaTeX) is real 0m2.730s user 0m1.836s sys 0m0.700s Sadly, I have no good way to see how much of this is caused by the callback, and how much is due to other LuaTeX particularities. However, one can run tex.tex through LuaTeX with and without this translation, and just run the normal TeX engine, too. tex tex.tex gives us real 0m1.253s user 0m0.496s sys 0m0.664s luatex tex.tex gives us real 0m14.329s user 0m13.413s sys 0m0.872s and using time luatex "&plain" '\directlua0{callback.register("process_input_buffer",function(buf)return unicode.utf8.char(unicode.latin1.byte(buf,1,-1))end)}\input tex' gives us real 0m14.801s user 0m12.709s sys 0m1.304s So the good news is that using the callback makes LuaTeX faster (more probably the difference gets lost in the noise). And the bad news is that it is about a factor of 25 slower than the normal TeX executable in either case. Some of it might be the difference in table sizes for the plain TeX executable. But the factor of 25 seems to fit rather well also with the LaTeX format test. Any idea where the bulk of this would be from? What would somebody wanting to use LuaTeX in a production environment do (apart from getting his head examined, I mean)? -- David Kastrup

3 23

buffer size musings
by David Kastrup 05 Apr '07

05 Apr '07

Hi, since I am off for Easter, I just want to spread the implementation idea for dealing with arbitrary long lines, so that I can get some feedback before I get to actual coding. Since LuaTeX has its own complications to take care off with regard to utf8, I actually would want to prepare this as a LuaTeX patch. Backporting to PDFTeX should be straightforward. I'll also try thinking about some input encoding implementation place: the process_input_buffer callback has several drawbacks: for one thing, it can't expect to get complete lines if I prepare lines peacemeal. So it would probably have to get another argument "partial". For another, the line end needs to get detected in the first place. But line detection _depends_ on the encoding, at least if we are talking about utf16 flavors. And in particular if process_input_buffer gets partial lines, I'd like it to get lines that don't contain partial utf-8 sequences. So I do see a need for some more stuff specific to LuaTeX, and I don't want to design something that would be hard to port to it. So much for some background. The basic design would be the following: buffer happens to be a single buffer. This could probably be chosen with a total size of 32k (naturally, people will disagree here, but that will stay configurable). Before reading material from a file, TeX will _start_ by placing the \endlinechar before any other material, as a fixed 4-byte long utf-8 sequence (alternatively, it may be stored as part of the file data structure): its setting at the time of reading the file needs to get preserved until we finally reach the end of the file. Then the next line gets read into buffer, either until end of line is reached, or until the buffer read limit (2k sounds reasonable, could conceivably be configurable since it influences things like maximum size of \csname ...\endcsname) is hit. When processing material, we usually check for end of line condition, anyway. When those checks turn out true, we do another check for "really" end of line or just end of the buffered part. If it is just the end of the buffered part, then sufficient material from the end of the buffered part is copied to the front of the buffered part, more stuff is read in according to the buffer read limit (and possibly tacking on the buffered end line character at the end), and we resume. "sufficient material" means the maximum of a) 40 characters (probably 160 bytes will do) of error context for the input line context part of error messages. b) if we are in the middle of scanning a control sequence name, the beginning of the control sequence. If this copying process would not result in any more available space (making it possible to actually read in new material), we get the dreaded buffer overflow. Basically this concept appears sound to me. It would, however, be strictly restricted to file reading. Things like \scantokens and \csname (which also use the buffer) would still be required to have their argument fit in one piece. But I guess that the file reading should cover the largest problem area. Something like that. I hope I'll have something to show before EuroTeX. But as I said, I am away without net access (and without computer) for the next week. All the best, David -- David Kastrup

3 6

\endlinechar can't be set to non-ASCII values.
by David Kastrup 04 Apr '07

04 Apr '07

Hi, the following program barfs and gets confused: \endlinechar128 \relax \relax \end This is luaTeX, Version 3.141592-snapshot-2007040210 (Web2C 7.5.6) (INITEX) (/tmp/junk.tex ! Buffer contains an invalid utf-8 sequence. l.2 \ relax ? ! Undefined control sequence. l.2 \rel ax ? ! Buffer contains an invalid utf-8 sequence. l.2 \relax ? ! Buffer contains an invalid utf-8 sequence. l.3 \ relax ? ! Undefined control sequence. l.3 \rel ax ? ! Buffer contains an invalid utf-8 sequence. l.3 \relax ? ! Buffer contains an invalid utf-8 sequence. l.4 \ end ? ! Undefined control sequence. l.4 \e nd ? ! Buffer contains an invalid utf-8 sequence. l.4 \end ? ) * ! Emergency stop. <*> /tmp/junk.tex No pages of output. Transcript written on junk.log. Maybe it would be better if buffer was an array of ASCII rather than packedASCII, namely encoded in UTF21 (or whatever) internally rather than UTF8. As it stands, dealing with \endlinechar and the end of buffer will not exactly be a lot of fun. Or one will need a function that is going to place a single character into buffer, and one needs to take care that at all times at least 4 (rather than one) characters remain free whenever \endlinechar has not yet been appended. -- David Kastrup

2 2

Proposed patch for the nc/nd problem
by David Kastrup 04 Apr '07

04 Apr '07

Seems like I can't get a patch attached to a bug report in the PDFTeX system, and Martin has closed the bug after fixing the first half. Here is a proposed patch to deal with the rest (note that 9.99998nd gets a wrong value, too, with the current code even though it adheres to Knuth's original description of the constraints). It also fixes Knuth's comment, and puts the nd ratio in shortest terms. The patch is based on LuaTeX, but it should be possible to apply it to pdfTeX. -- David Kastrup

2 1

Just noticed something...
by David Kastrup 04 Apr '07

04 Apr '07

In the messages from the tex.tex run, I got overfull box messages like the following: Overfull \hbox (12.40152pt too wide) in paragraph at lines 21018--21025 []\tenbf 836. \tenrm It is not nec-es-sary to cre-ate new ac-tive nodes hav-ing [] greater than $[] +| \hbox(6.94444+1.94444)x469.75499, glue set - 1.0, direction TLT .\whatsit ..\localinterlinepenalty=0 ..\localbrokenpenalty=0 ..\localleftbox=null ..\localrightbox=null .\tenbf 8 .\tenbf 3 .\tenbf 6 .\tenbf . .etc. That whatsit has to go. Absolutely. When the user is explicitly assigning values to \local* variables, placing a whatsit at the point of assignment is fine and expected. It is even conceivable to place a whatsit at the start of the current hlist _when_ we assign to \local* in the middle of the list. But we can't go peppering the lists with whatsits when we are not asked for it. It will break far too many existing packages. Unless we find a way to make whatsits transparent. But the combination glue whatsit glue could not preserve a whatsit when doing \skip0=\lastskip \unskip \skip1=\lastskip \unskip \vskip\skip1 \vskip\skip2 So there is -- David Kastrup, Kriemhildstr. 15, 44793 Bochum

3 2

A design philosophy question...
by David Kastrup 03 Apr '07

03 Apr '07

I have this year reported something as a bug to Barbara Beeton (the bug collector for TeX) which I find disturbing: TeX's arithmetic on units truncates and thus is biased. This has the annoying effect that "1in", supposedly representing one inch which is _exactly_ 72.27 points, will in TeX stand for the value "72.26999pt" and be actually smaller than the value "72.27pt". Of course, fixing this might change existing documents. Also we have: **\dimen0=1in entering extended mode *\showthe\dimen0 > 72.26999pt. *\showthe\dimexpr100\dimen0\relax > 7226.9989pt. *\dimen0=72.27pt *\showthe\dimen0 > 72.27pt. *\showthe\dimexpr100\dimen0\relax > 7227.00043pt. So while "72.27pt" is _quite_ closer to one inch than "1in" is, it is larger. Anyway, is something like that an area that LuaTeX would actually ever consider touching? After all, it would break document compatibility. Which might be, I believe, less of a holy grail for ConTeXt than for LaTeX. Opinions? -- David Kastrup, Kriemhildstr. 15, 44793 Bochum

4 12

What is the unit "nc" supposed to be?
by David Kastrup 03 Apr '07

03 Apr '07

New Cicero or what? Please consider the following: This is luaTeX, Version 3.141592-snapshot-2007040210 (Web2C 7.5.6) **\dimen0=1nc *\showthe\dimen0 > 12.80373pt. <*> \showthe\dimen0 ? *\dimen0=0.99998nc *\showthe\dimen0 > -0.09723pt. <*> \showthe\dimen0 ? * This does not cut it at all. The section in tex.web starts with @ The necessary conversion factors can all be specified exactly as fractions whose numerator and denominator are 32768 or less. In LuaTeX, the section brazenly starts with @ The necessary conversion factors can all be specified exactly as fractions whose numerator and denominator are 65535 or less. Who does things like that without actually checking that they work? It is possible to rework the arithmetic to get along with 65535, but it is certainly not a matter of merely changing the _documentation_ and not the code. I have no idea whether this is an eTeX, a PDFTeX, an Aleph or a LuaTeX invention (did I forget anything?), but it certainly can't stay like that. Sorry if I sound like a whiner here. If you think it worth the trouble, I can rework the arithmetic (but there is a reason that Knuth kept to 15 bit factors, so it is going to become more complicated). But I consider it more likely that nc should be changed to 1370:107 which is the closest 15 bit fraction to 12nd (it is smaller than 12nd, but the next larger suitable fraction 32163:2512 is slightly further from 12nd). Please remember that in TeX, 2in is not equal to 1in+1in anyway. I think we can afford this. Certainly better than the current behavior. -- David Kastrup

4 11

Rationale for using putenv?
by David Kastrup 02 Apr '07

02 Apr '07

Hi, in luastuff.c we have static int os_setenv (lua_State *L) { char *value, *key, *val; key = (char *)luaL_optstring(L, 1, NULL); val = (char *)luaL_optstring(L, 2, NULL); if (key) { if (val) { value = xmalloc(strlen(key)+strlen(val)+2); sprintf(value,"%s=%s",key,val); if (putenv(value)) { return luaL_error(L, "unable to change environment"); } } else { #if defined(WIN32) || defined(__sun__) value = xmalloc(strlen(key)+2); sprintf(value,"%s=",key); if (putenv(value)) { return luaL_error(L, "unable to change environment"); } #else (void)unsetenv(key); #endif } } lua_pushboolean (L, 1); return 1; } Now putenv is not required by Posix, as opposed to setenv. The main rationale for "putenv" is that it leaves memory management of the string to the application, and thus can avoid leaking memory. However, the above code just uses xmalloc without deallocating previously set values, so it leaks, anyway. Basically I see two ways of dealing with this: a) maintain a table of key -> "key=value" mappings for environment variables that have been set by Lua. That way, one can keep track of what strings need to stay allocated, and which can be deallocated again. b) don't bother about the leak and use setenv instead. Option b) would be somewhat more portable, I believe. Of course, if people actually use os_setenv to switch around between environment variables frequently, the leak might get annoying at some point of time. -- David Kastrup

3 10