Hi, Hans: in font-afm.lua, you use fontforge to open and to_table a pfb, but without closing it? btw, when MKIV is running for the first time (i.e., no cache file). it will use a lot of memory (parsing a 10M font will use 700-800 MB memory), are there some ways to reduce that? Another problem is, the memory usage seems incremental: I watch the memory usage in "top" carefully, it does not reduce when MKIV finish reading and caching an otf font. (well I read the font-otf.lua, and you are actually closing that font. so maybe that's because luatex is leaking memory?) btw, I load arial 300 times using the following file, %engine=luatex \directlua0{ function load_font (filename) local metrics = nil local font = fontforge.open(filename) if font then metrics = fontforge.to_table(font) fontforge.close(font) end return metrics end myfont = load_font('arial.ttf') % 300 myfont = load_font('arial.ttf') myfont = load_font('arial.ttf') myfont = load_font('arial.ttf') myfont = load_font('arial.ttf') myfont = load_font('arial.ttf') } and I can see the memory usage may ebb and flow, but actually it is leaking memory. Yue Wang
Yue Wang wrote:
Hi, Hans:
in font-afm.lua, you use fontforge to open and to_table a pfb, but without closing it?
ok, thanks for noticing, i added a fontforge.close(pfbblob)
btw, when MKIV is running for the first time (i.e., no cache file). it will use a lot of memory (parsing a 10M font will use 700-800 MB
hm so much?
memory), are there some ways to reduce that? Another problem is, the memory usage seems incremental: I watch the memory usage in "top" carefully, it does not reduce when MKIV finish reading and caching an otf font. (well I read the font-otf.lua, and you are actually closing that font. so maybe that's because luatex is leaking memory?)
it's hard to say ... it related to the lua garbage collector i already discard the loaded table and reload from cache (which save mem) but one has to wait till the garbage collector kicks in which can later also, mem consumption reported is a bit of a lie because lua seems to allocates double of what it needed with each increment but not always use it otf fonts are already quite packed in mkiv (i'm considering adding an option to mtx-font to precache fonts but i'm not sure if it's worth the trouble is mem consumption less a second run? we use our mk document as test and there mem consumption is ok (given the number of loaded fonts) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
ok, thanks for noticing, i added a
fontforge.close(pfbblob)
good
btw, when MKIV is running for the first time (i.e., no cache file). it will use a lot of memory (parsing a 10M font will use 700-800 MB
hm so much?
Yes, when finish reading the following tex file without \bye using plain TeX, LuaTeX will cost 719M memory on my machine. The font used is Adobe song std light. [yue@localhost ~]$ cat a.tex \directlua0{ function load_font (filename) local metrics = nil local font = fontforge.open(filename) if font then metrics = fontforge.to_table(font) fontforge.close(font) end return metrics end myfont = load_font('song.otf') } [yue@localhost ~]$ when using the above script loading four fonts (you can try this by adding 3 myfont = load_font('song.otf')), the memory consumption will be higher than 1G on my machine. Remember I am using Plain TeX to do this test, MKIV will consume more memory.
also, mem consumption reported is a bit of a lie because lua seems to allocates double of what it needed with each increment but not always use it
otf fonts are already quite packed in mkiv (i'm considering adding an option to mtx-font to precache fonts but i'm not sure if it's worth the trouble
is mem consumption less a second run?
Yes, dramatically less. with 4 chinese fonts loaded (I use zhfonts.tex for testing), MKIV consumes 300-400MB. Yue Wang
Yue Wang wrote:
also, mem consumption reported is a bit of a lie because lua seems to allocates double of what it needed with each increment but not always use it
otf fonts are already quite packed in mkiv (i'm considering adding an option to mtx-font to precache fonts but i'm not sure if it's worth the trouble
is mem consumption less a second run?
Yes, dramatically less. with 4 chinese fonts loaded (I use zhfonts.tex for testing), MKIV consumes 300-400MB.
on my machine running mk with a fresh cache gives the same lua mem usage as with filled cache, however a fresh run tops at 1 meg while a filled run tops ar 500 (windows vista); that document uses several cjk fonts (relatively large due to many glyphs) and script fonts (some arabic and zapfino as well as cambria math which are much larger fonts due to many features mkiv lua stats : loaded fonts - adobesongstd-light:otf arabtype:ttf c:/windows/fonts/cambria:ttc htfs:ttf htsong:ttf ko eieletters:afm lmex10:tfm lmmi10:tfm lmmib10:tfm lmmono10-regular:otf lmmonoltcond10-regular:otf lmroman10-italic:otf lmroman10-regu lar:otf lmroman12-regular:otf lmsans10-regular:otf lmsy10:tfm lmtypewriter10-regular:otf msam10:tfm msbm10:tfm palatinoltarabic-regu lar:ttf rm-lmbx10:tfm rm-lmr10:tfm rm-lmr12:tfm rm-lmr8:tfm rm-lmtt10:tfm scheherazaderegot:ttf texnansi-lt55484:afm texnansi-lt5548 5:afm texnansi-lt55488:afm texnansi-lt55507:afm zapfinoextraltpro:otf mkiv lua stats : current memory usage - 361 MB (ctx: 370 MB) (keep in mind that at some point luatex has to load the glyph shapes and pass them to the pdf file so there will always be otf/pfb loading at the end of a run) as taco mentioned, real memory usage for a fresh run is probably somewhat lower due to fragmentation ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
on my machine running mk with a fresh cache gives the same lua mem usage as with filled cache, however a fresh run tops at 1 meg while a filled run tops ar 500 (windows vista); that document uses several cjk fonts (relatively large due to many glyphs) and script fonts (some arabic and zapfino as well as cambria math which are much larger fonts due to many features
mkiv lua stats : loaded fonts - adobesongstd-light:otf arabtype:ttf c:/windows/fonts/cambria:ttc htfs:ttf htsong:ttf ko eieletters:afm lmex10:tfm lmmi10:tfm lmmib10:tfm lmmono10-regular:otf lmmonoltcond10-regular:otf lmroman10-italic:otf lmroman10-regu lar:otf lmroman12-regular:otf lmsans10-regular:otf lmsy10:tfm lmtypewriter10-regular:otf msam10:tfm msbm10:tfm palatinoltarabic-regu lar:ttf rm-lmbx10:tfm rm-lmr10:tfm rm-lmr12:tfm rm-lmr8:tfm rm-lmtt10:tfm scheherazaderegot:ttf texnansi-lt55484:afm texnansi-lt5548 5:afm texnansi-lt55488:afm texnansi-lt55507:afm zapfinoextraltpro:otf
depends on how you use them. \font\a=simsun at 10pt and \definetypeface blablabla [simsun] are different.
mkiv lua stats : current memory usage - 361 MB (ctx: 370 MB)
You cannot depend on the memory report by mtxrun. The actual memory usage is *much* larger than the reported value. Only the one reported by the system is believable.
(keep in mind that at some point luatex has to load the glyph shapes and pass them to the pdf file so there will always be otf/pfb loading at the end of a run)
as taco mentioned, real memory usage for a fresh run is probably somewhat lower due to fragmentation
The memory usage without taking fragmentation into account has nothing to do with the user's experience. With 1G memory on my machine, and the best memory allocation implementation I can find (The FreeBSD operating system's memory allocation algorithm is very efficient. Firefox 3 also uses its implementation). In order to use ConTeXt with 4 chinese fonts, I should - First, quit Firefox, Gnome and X Window - Run sudo shutdown now and log into the single user mode - Run context foobar.tex, when finishing dumping the first Chinese font's font cache, press control + C, and run that again for the second font. And that's my experience. If I don't do that, the swap will be caching for several minutes, and finally the kernel stopped the luatex program since it uses too much memory.
----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl ----------------------------------------------------------------- ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net ___________________________________________________________________________________
Yue Wang wrote:
And that's my experience. If I don't do that, the swap will be caching for several minutes, and finally the kernel stopped the luatex program since it uses too much memory.
Can you send me that input file and the fonts it needs (if they are not freely available)? Best wishes, Taco
Hi Yue Wang, Yue Wang wrote:
myfont = load_font('arial.ttf') myfont = load_font('arial.ttf') } and I can see the memory usage may ebb and flow, but actually it is leaking memory.
I also see a non-trivila leak (somewhere inside the fontforge library). I will investigate further next week, it is not that massive. I get this result for your input file, with 500 arial.ttfs (ArialMT version 3.0, 367112 bytes): ==8819== LEAK SUMMARY: ==8819== definitely lost: 16,965,191 bytes in 707,541 blocks. ==8819== possibly lost: 1,128 bytes in 22 blocks. ==8819== still reachable: 202,135,427 bytes in 81,127 blocks. ==8819== suppressed: 0 bytes in 0 blocks. ==8819== Rerun with --leak-check=full to see details of leaked memory. So some 32kb per font. Noticeable, but not enormous. The growth in 'top' is probably caused more by memory fragmentation than by this leak. Best wishes, Taco
participants (3)
-
Hans Hagen -
Taco Hoekwater -
Yue Wang