Hello Thomas,

do you know the article "breaking paragraph into lines"
by knuth et. al.? It can be purchased at wiley:
http://onlinelibrary.wiley.com/doi/10.1002/spe.4380111102/abstract

Also it is contained as chapter 3 in Knuths book "Digital Typography".
http://www-cs-faculty.stanford.edu/~uno/dt.html

Another interestung source for you could be this:
https://github.com/simoncozens/sile
He created a kind of TeX-clone in Lua.

--
Mit freundlichen Grüßen
Christof Schardt
mailto:christof@schardt.info