Hi to all Apologies in advance for the slightly off-topic nature of this post. I've been reading about HarfBuzz which certainly looks to be an interesting project. From the project page there is a link to an article titled "State of Text Rendering" by Behdad Esfahbod http://behdad.org/text/ Within that article is a section on TeX engines with a reference to XeTeX and LuaTeX --- I'm just quoting a small bit of the article which I hope is not too much out of context: "The XeTeX creator is working on HarfBuzz on behalf of Mozilla now, and plans to port XeTeX to HarfBuzz eventually. In the long term though, pdfTeX's successor luaTeX should be made to do the same thing. There is more to Unicode support than just shaping, and in those areas the TeX engines can gain a lot by building on top of existing libraries." At present my own understanding of the precise role that HarfBuzz could play is a little sketch/hazy right now, but I'm working on that ;-). In the meantime I'd be interested to know if, in theory, HarfBuzz could be usefully plugged into LuaTeX, such as via an external library (.so/.dll).From what I understand... HarfBuzz provides a shaping engine so that, for example, it could provide useful services such as the bidi algorithm, and information that could be useful for typesetting complex scripts such as Arabic. Again, I stress my currently incomplete understanding of the precise services HarfBuzz would provide. lso, how would you pass back to LuaTeX the results so that the LuaTeX engine could use them? If anyone could help me understand this a little more or indeed let me know if there really is nothing to explore that too would be good. I have not tried to compile HarfBuzz yet so have not played with it at all. I'm merely curious in light of the remarks on the HarfBuzz site. Thank to anyone who can shed some light on this for me. Cheers Graham
Le 01/02/2011 20:21, Graham Douglas a écrit :
Hi to all
Apologies in advance for the slightly off-topic nature of this post. I've been reading about HarfBuzz which certainly looks to be an interesting project. From the project page there is a link to an article titled "State of Text Rendering" by Behdad Esfahbod
Within that article is a section on TeX engines with a reference to XeTeX and LuaTeX --- I'm just quoting a small bit of the article which I hope is not too much out of context:
"The XeTeX creator is working on HarfBuzz on behalf of Mozilla now, and plans to port XeTeX to HarfBuzz eventually. In the long term though, pdfTeX's successor luaTeX should be made to do the same thing. There is more to Unicode support than just shaping, and in those areas the TeX engines can gain a lot by building on top of existing libraries."
At present my own understanding of the precise role that HarfBuzz could play is a little sketch/hazy right now, but I'm working on that ;-). In the meantime I'd be interested to know if, in theory, HarfBuzz could be usefully plugged into LuaTeX, such as via an external library (.so/.dll).From what I understand... HarfBuzz provides a shaping engine so that, for example, it could provide useful services such as the bidi algorithm, and information that could be useful for typesetting complex scripts such as Arabic. Again, I stress my currently incomplete understanding of the precise services HarfBuzz would provide. lso, how would you pass back to LuaTeX the results so that the LuaTeX engine could use them?
If anyone could help me understand this a little more or indeed let me know if there really is nothing to explore that too would be good. I have not tried to compile HarfBuzz yet so have not played with it at all. I'm merely curious in light of the remarks on the HarfBuzz site.
I don't understand exactly what HarfBuzz does, but as I get it it maps characters to glyphs, what it calls "text shaping", taking care of such things as substitutions and the like. With LuaTeX (which I don't pretend to understand exactly either), when it comes to such matters, you're alone: you're the one in charge of "text shaping", even for ligatures from pairs of glyphs (e.g. "ff"), since you're supposed to have created them when loading the font. So I don't think integrating HarfBuzz would be in line with LuaTeX's philosophy, otherwise it would be no different from XeTeX in that respect. On the other hand, nothing prevents you (the user) from calling HarfBuzz (or doing whatever you're supposed to do with it) from LuaTeX, I guess. As for the sentence "LuaTeX should be made to do the same thing", if "should" means "it'd be good", I suppose the author hasn't really understood LuaTeX; if "should" means "it is going to happen, normally", then /I/ haven't really understood LuaTeX. What the last sentence means is obscure to me. Again, I don't know much about HarfBuzz, and I can't obviously make any authorized statement on LuaTeX, but that's how I see things. Best, Paul
On 1-2-2011 8:54, Paul Isambert wrote:
With LuaTeX (which I don't pretend to understand exactly either), when it comes to such matters, you're alone: you're the one in charge of "text shaping", even for ligatures from pairs of glyphs (e.g. "ff"), since you're supposed to have created them when loading the font. So I don't think integrating HarfBuzz would be in line with LuaTeX's philosophy, otherwise it would be no different from XeTeX in that respect. On the other hand, nothing prevents you (the user) from calling HarfBuzz (or doing whatever you're supposed to do with it) from LuaTeX, I guess.
As for the sentence "LuaTeX should be made to do the same thing", if "should" means "it'd be good", I suppose the author hasn't really understood LuaTeX; if "should" means "it is going to happen, normally", then /I/ haven't really understood LuaTeX. What the last sentence means is obscure to me.
Again, I don't know much about HarfBuzz, and I can't obviously make any authorized statement on LuaTeX, but that's how I see things.
LuaTeX is and will remain a TeX engine, an opened up one. One of the principles is that we won't hard code solutions other than the one provided by (traditional) TeX (although unicode is the coding and wide tfm fonts are corner stones). There is the default (relatively lightweight) font loader based on fontforge. There is no universal solution and hardcoding or binding to some other library is no option, if only because they come and go, and fashion can change. The idea is to have a stable engine (as stable as traditional tex itself) that can act as basis. However, as one can load Lua libraries, any library that has a Lua interface can in principle be plugged in. So if someone wants another renderer, he/she can plug in something and as long as what gets returned suits tex's inner model (node lists) it's ok. In that respect luatex is indifferent to how/what/where it happens. Using some renderer is not so much different from using libraries to access databases or filesystems or ... So, to answer your question: no extrnal rendering libraries will be linked in by default and none will be provided by the luatex team, but one can imagine external libraries to be used that get fed data from luatex and give something back that gets fed back into the machinery. Of course, when a macro package depends on such libraries, it adds another depedency which can be a long term maintainance problem. Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
Hi Graham, very interesting link/thought. Thanks. TeX has been the software for the best typesetting, at least for english texts and for some other languages and scripts. Now with XeTeX and LuaTeX we have the chance to extend the covered scripts to all used ones, so I guess. So we need a text shaping engine, there is no doubt about that. The current situation is that every macro package has to reinvent the wheel and with LuaTeX, there will be many more applications beyond classic \TeX input, and these application also need to reinvent the wheel. So having a text shaping engine such as harfbuzz is only a question of time. I'd like to see LuaTeX shipping with some default set of modules, for example an harfbuzz or an xml parser (I still can't believe that there is none built in)(, such as TeXlive ships with a bunch of LaTeX packages). Each one with a suitable interface for LuaTeX. I don't expect this to happen anytime soon, for LuaTeX still needs a wider spread audience and there is currently a showstopper that prevents module loading in LuaTeX, but one day some experts will jump in and provide glue code/modules. If LuaTeX does not open up, there will be a risk of forks. There is already a LuaTeX fork with an xml parser built in. I can understand the reluctance of the LuaTeX team to include unnecessary modules, and it would be helpful if there was a "contrib" directory with third party modules that will be built as a library (.so, .dll) which get installed in the TeXlive system. So one can say for example 'require "harfbuzz"' and have harfbuzz loaded on all platforms LuaTeX is running on. Patrick
On 2-2-2011 8:11, Patrick Gundlach wrote:
If LuaTeX does not open up, there will be a risk of forks. There is already a LuaTeX fork with an xml parser built in. I can understand the reluctance of the LuaTeX team to include unnecessary modules, and it would be helpful if there was a "contrib" directory with third party modules that will be built as a library (.so, .dll) which get installed in the TeXlive system. So one can say for example 'require "harfbuzz"' and have harfbuzz loaded on all platforms LuaTeX is running on.
Eh ... luatex is opened up. One can just load libraries. Preloading a whole lot of (arbitrary) libraries makes no sense. I can imagine that at some point (when luatex 1 is out) that there will be a bunch of libs shipped alongside the binary distributions but that then has to be part of regular tex live and miktex and ... distributions. The thing with libraries is that they evolve independently and you cannot expect the luatex team to keep track of them. The only thing we can provide is a stable api into luatex. It's not a matter of opening up, but a matter of packaging. Also, when I run luatex I don't want at some point in the future to find out that an additional 125 libs (plus dependencies) are needed that I don't use. A repository with additional modules is a secondary effort, alongside luatex. I know that Luigi has done lots of tests and is setting up some procedure for it, but as this is to be supported on *all* relevant platforms it's not something trivial. So, there will be something, but not preloaded. (Just like we don't preload macro definitions or fonts or ...). Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
On Wed, Feb 2, 2011 at 9:30 AM, Hans Hagen
A repository with additional modules is a secondary effort, alongside luatex. I know that Luigi has done lots of tests and is setting up some procedure for it, but as this is to be supported on *all* relevant platforms it's not something trivial. So, there will be something, but not preloaded. (Just like we don't preload macro definitions or fonts or ...).
Hans
True. There are some solutions for this 1) handmade binding The most efficient and long 2) SWIG Automatic binding. Quick, solid, but you need a very good C library. C++ is a nightmare (at least for me now) 3) a bridge ala lunatic. If a python binding already exist, why build a different lua binding ?. Just use a bit of lua/python mixed code. Of course there are and will be always problems for maintenance (think about python 2.6,2.7,3.1,3.2 for example) For all these points, support for all platforms is something hard to achieve. Actually I can compile and test in (in a 32bit Linux box) libraries for Linux and Windows with mingw but I don't have any clue if I can do the same with Mac (of course without buy a Mac) Concerning xml: build a binding for libxml & libxslt with swig should be easy; the API is stable, the library is well tested and available for most platform. Perhaps already exists a lua binding and even an handmade binding is feasible --- so why a fork of Luatex ? -- luigi
Concerning xml: build a binding for libxml & libxslt with swig should be easy; the API is stable, the library is well tested and available for most platform. Perhaps already exists a lua binding and even an handmade binding is feasible --- so why a fork of Luatex ?
Because a) loading such a module into existing luatex binaries is impossible at the moment (see other mail), b) shipping one platform dependent file instead of two is less hassle and c) compiling only one source is much easier then compiling two or more sources. Patrick
On Wed, Feb 2, 2011 at 10:03 AM, Patrick Gundlach
Concerning xml: build a binding for libxml & libxslt with swig should be easy; the API is stable, the library is well tested and available for most platform. Perhaps already exists a lua binding and even an handmade binding is feasible --- so why a fork of Luatex ?
Because a) loading such a module into existing luatex binaries is impossible at the moment (see other mail), b) shipping one platform dependent file instead of two is less hassle and c) compiling only one source is much easier then compiling two or more sources.
Too much drastic. Actually it's possibile to load non-trivial library but not *all* library. As you wrote in #555 with sqlite there are some problems, and I believe that with SWIG you can solve them *without* touch the luatex source/building phase
b) shipping one platform dependent file instead of two is less hassle and c) compiling only one source is much easier then compiling two or more sources. Again shipping and compiling maybe can be better, but not maintenance. A bug in xyz library will be a bug in luatex that the luatex team must solve for all platform.
Even for libxml thare are some prons & cons: a xml-based workflow probably doesn't need the xml side of luatex. -- luigi
Am 02.02.2011 um 09:30 schrieb Hans Hagen:
On 2-2-2011 8:11, Patrick Gundlach wrote:
If LuaTeX does not open up, there will be a risk of forks. There is already a LuaTeX fork with an xml parser built in. I can understand the reluctance of the LuaTeX team to include unnecessary modules, and it would be helpful if there was a "contrib" directory with third party modules that will be built as a library (.so, .dll) which get installed in the TeXlive system. So one can say for example 'require "harfbuzz"' and have harfbuzz loaded on all platforms LuaTeX is running on.
Eh ... luatex is opened up. One can just load libraries.
No. Currently it is impossible to take a LuaTeX binary (from TeXlive for example) and load any non-trivial c-based module. At least for Mac OS X but I guess this is the same on all unix platforms. This makes it impossible to package for example an xml reader lua module into texlive. (See tracker item #555.) But this is - I assume - just a matter of time... I am _not_ asking for including a whole bunch of libraries into LuaTeX. And I don't expect the already burdened LuaTeX (core) team to do all the work including the libraries. But the community could provide something like tlcontrib or modules.contextgarden.net where we collect luatex'ed libraries that eventually CTAN picks up and include into texlive/texmf/libs/luatex/ (or whatever). Or the mentioned contrib directory in the luatex source, where the optional modules are. There will of course be complaints such as "my LaTeX document foo.tex does not work together with the library bar.dll in LuaTeX" as we have packages clash in LaTeX today. That's the backside of openness.
[...] and you cannot expect the luatex team to keep track of them. The only thing we can provide is a stable api into luatex.
I don't expect this and the [stable] API is already there. I don't mind LuaTeX changing its API as long as it is documented somewhere (in the changelogs for example). Patrick
Just to clarify what I mean:
If LuaTeX does not open up, there will be a risk of forks. There is already a LuaTeX fork with an xml parser built in. I can understand the reluctance of the LuaTeX team to include unnecessary modules[...]
with that I mean a) the technical base of loading modules and b) the community to jump in and provide bindings which will be packed in a distribution. I can see (and I agree) why additional libraries are not hard coded into the luatex core. So I didn't want to criticize the LuaTeX team in any way and if I did, I apologize. Patrick
On Tue, Feb 01, 2011 at 07:21:44PM +0000, Graham Douglas wrote:
Hi to all
Apologies in advance for the slightly off-topic nature of this post. I've been reading about HarfBuzz which certainly looks to be an interesting project. From the project page there is a link to an article titled "State of Text Rendering" by Behdad Esfahbod
Within that article is a section on TeX engines with a reference to XeTeX and LuaTeX --- I'm just quoting a small bit of the article which I hope is not too much out of context:
"The XeTeX creator is working on HarfBuzz on behalf of Mozilla now, and plans to port XeTeX to HarfBuzz eventually. In the long term though, pdfTeX's successor luaTeX should be made to do the same thing. There is more to Unicode support than just shaping, and in those areas the TeX engines can gain a lot by building on top of existing libraries."
At present my own understanding of the precise role that HarfBuzz could play is a little sketch/hazy right now, but I'm working on that ;-). In the meantime I'd be interested to know if, in theory, HarfBuzz could be usefully plugged into LuaTeX, such as via an external library (.so/.dll).From what I understand... HarfBuzz provides a shaping engine so that, for example, it could provide useful services such as the bidi algorithm, and information that could be useful for typesetting complex scripts such as Arabic. Again, I stress my currently incomplete understanding of the precise services HarfBuzz would provide. lso, how would you pass back to LuaTeX the results so that the LuaTeX engine could use them?
If anyone could help me understand this a little more or indeed let me know if there really is nothing to explore that too would be good. I have not tried to compile HarfBuzz yet so have not played with it at all. I'm merely curious in light of the remarks on the HarfBuzz site.
Thank to anyone who can shed some light on this for me.
HarfBuzz provide an OpenType layout engine (Graphite support existed at some point, but were removed to be rewritten, AAT is planned in some distant future) but no bidi (the client have to do it in its own or use other libraries like fribidi). HarfBuzz workflow is usually 1) loading a font in FreeType 2) shaping text in HarfBuzz for that font 3) getting back glyph indices and positions. Of course there are more details like itemizing the text, selecting font features and so on. FreeType is not mandatory, but using it simplifies the process. Hans have written (and still improving) an OpenType layout engine completely in lua(tex) and it more or less does what HarfBuzz would do, it may support more features than HarfBuzz but HarfBuzz code is more widely tested, so each have its pros and cons. I can imagine a separate module where HarfBuzz is plugged into LuaTeX allowing it to process node lists instead of text strings (HarfBuzz have callbacks that I think can be used to plug function that deal with node buffers instead of text strings), it can be an interesting exercise and I expect HarfBuzz maintainer to be willing to cooperate. Regards, Khaled -- Khaled Hosny Egyptian
HarfBuzz provide an OpenType layout engine (Graphite support existed at some point, but were removed to be rewritten, AAT is planned in some distant future) but no bidi (the client have to do it in its own or use other libraries like fribidi).
HarfBuzz workflow is usually 1) loading a font in FreeType 2) shaping text in HarfBuzz for that font 3) getting back glyph indices and positions. Of course there are more details like itemizing the text, selecting font features and so on.
FreeType is not mandatory, but using it simplifies the process.
Hans have written (and still improving) an OpenType layout engine completely in lua(tex) and it more or less does what HarfBuzz would do, it may support more features than HarfBuzz but HarfBuzz code is more widely tested, so each have its pros and cons.
I can imagine a separate module where HarfBuzz is plugged into LuaTeX allowing it to process node lists instead of text strings (HarfBuzz have callbacks that I think can be used to plug function that deal with node buffers instead of text strings), it can be an interesting exercise and I expect HarfBuzz maintainer to be willing to cooperate.
Regards, Khaled
--
Khaled Hosny Egyptian
Dear Khaled Thanks very much indeed for this helpful summary. I've now compiled HarfBuzz on Windows (using Visual Studio) so, it's time to explore it :-) Just to add, I do of course understand why libraries such as HarfBuzz should not be part of core LuaTeX and only ever "plugged in" via library facilities as a DLL/so. As I'm sure many others have experienced when using/building tools, it can be quite annoying to hit "multiple cascading library dependancies" which can be a real nightmare to sort out even for a one-off build, let alone trying to maintain! Thanks again to everyone for the discussions on this thread. Cheers Graham
participants (6)
-
Graham Douglas -
Hans Hagen -
Khaled Hosny -
luigi scarso -
Patrick Gundlach -
Paul Isambert