Re: [NTG-context] Count (and limit) glyphs per line?

26 Jun 2022

      Hi Max, 

Thank you so much for your help and pointing me to the documents; always a lot of things to learn in TeX!

I'm afraid that including the hyphen width doesn't solve the issue yet. It seems to move the problem to other parts of the text. My guess is that one could equivalently have said "local max_length = 111", right?

I made the following MWE (reproducible also online) to illustrate what I see: 

* Here, instead of a breaking point, the trouble is caused by not being able to break it. This causes the next line to be underfull. (I get a lot of these, but also some with hyphenated breakpoints, in my own document. Maybe the insertion point of the penalty/breaking bonus needs to move up?) 

* Running with hsize only makes the problem worse in itemizations, so I think localhsize is the way to go. My guess, localhsize is the width of the "text" part of a paragraph, for example, excluding the symbols in the itemization.

(More thoughts below)

\startluacode
       local max_length = 112

       local glyph_id = node.id "glyph"
       local disc_id = node.id "disc"
       local glue_id = node.id "glue"

       function userdata.limiter(head)
           head = language.hyphenate(head)

           local hyphen = node.new "glyph"
           hyphen.char = language.prehyphenchar(0)
           hyphen.font = font.current()
           local width = hyphen.width
           node.free(hyphen)

           local chars = 0
           local n = head
           while n do
               if n.id == glyph_id or n.id == glue_id then
                   chars = chars + 1
                   width = width + n.width - (n.shrink or 0)
               end

               local localhsize = tex.dimen["textwidth"]

               if tex.dimen["localhsize"] > 0 then
                   localhsize = tex.dimen["localhsize"]
               end

               if chars >= max_length or width > localhsize then
                   local back_chars = 0
                   local end_disc = nil

                   while n do
                       if n.id == glue_id then
                           local penalty = node.new "penalty"
                           penalty.penalty = -10000
                           node.insertbefore(head, n, penalty)
                           break
                       end

                       if not end_disc and n.id == disc_id then
                           end_disc = n
                       end

                       if end_disc and back_chars >= 5 then
                           end_disc.penalty = -10000
                           break
                       end

                       if n.id == glyph_id then
                           back_chars = back_chars + 1
                       end

                       n = n.prev
                   end

                   width = 0
                   chars = 0
               end

               n = n.next
           end

           return head
       end

       nodes.tasks.appendaction(
           "processors",
           "before",
           "userdata.limiter"
       )
   \stopluacode

   \setuppapersize[A5]
   \showframe

   \starttext

		This is text width:
	   \ctxlua{context(tex.dimen["textwidth"])}

	   This is hsize:
	   \ctxlua{context(tex.dimen["hsize"])}

	   This is localhsize:
	   \ctxlua{context(tex.dimen["localhsize"])}

	   \startitemize[width=5em]

	   \item Thus, I came to the conclusion that the \hbox{designer} of a new system must not only be the implementer and first large--scale user; the de signer should also write the first user manual.

	   \item \samplefile{knuth}

		This is text width:
	   \ctxlua{context(tex.dimen["textwidth"])}

	   This is hsize:
	   \ctxlua{context(tex.dimen["hsize"])}

	   This is localhsize:
	   \ctxlua{context(tex.dimen["localhsize"])}

	   \stopitemize

   \stoptext

I'm wondering if I do understand the second while loop correctly:

* Once we find the node that exceeds either the character limit or the (local-)hsize (glyphs and glues summed, for disc we add hyphen.width, do we?), then we insert an incredibly good breaking point for a new line. And exit the loop.

* The other cases still seem a bit obscure to me, and I tried to trace where each of them might be triggered:

                       if n.id == glue_id then
                           local penalty = node.new "penalty"
                           penalty.penalty = -10000
                           node.insertbefore(head, n, penalty)
						   context.inrightmargin("glue")
                           break
                       end

                       if not end_disc and n.id == disc_id then
					   	   context.inrightmargin("disc")
                           end_disc = n
                       end

                       if end_disc and back_chars >= 5 then
					       context.inrightmargin("end")
                           end_disc.penalty = -10000
                           break
                       end

                       if n.id == glyph_id then
                           context.inrightmargin("glyph")
						   back_chars = back_chars + 1
                       end

I'm maybe doing this wrong, but I see these conditions triggered more often than probably expected for a 25 line document? 

    local count_me = 0 

    ...

    if chars >= max_length or width > localhsize then
                   local back_chars = 0
                   local end_disc = nil

                   while n do
                           local check = "glyph"

                        count_me = count_me + 1
                       if n.id == glue_id then
                           local penalty = node.new "penalty"
                           penalty.penalty = -10000
                           node.insertbefore(head, n, penalty)
                           context.inrightmargin("\\color[red]{" .. string.rep("_", count_me) .. count_me .. "}")
                           break
                       end

                       if not end_disc and n.id == disc_id then
                           end_disc = n
                       end

                       --
                       if end_disc and back_chars >= 5 then
                           context.inrightmargin("\\color[blue]{" .. string.rep("_", count_me) .. count_me .. "}")
                           end_disc.penalty = -10000
                           break
                       end

                       if n.id == glyph_id then
                           context.inrightmargin("\\color[black]{" .. string.rep("_", count_me) .. count_me .. "}")
                           back_chars = back_chars + 1
                       end

                       n = n.prev
                   end

Many thanks again!

Benjamin
...
On Jun 25, 2022, at 17:40, Max Chernoff  wrote:
...
It's also a very insightful example of how to use and inject Lua code in the TeX output routine.
This is injecting Lua code before the paragraph builder, not in the output routine. Something like https://tex.stackexchange.com/a/644613/270600 or my module "lua-widow-control" would be an example of Lua code in the output routine.
...
Do you mind if I add it to the wiki? (Probably under "Wrapping".)
Sure
...
However, tex.localhsize (or tex.dimen["localhsize"]) is 0 when the document is initialized. (Maybe a more sensible default would be textwidth rather than 0?)
So, I added:
  local localhsize = tex.dimen["textwidth"]
if tex.dimen["localhsize"] > 0 then
      localhsize = tex.dimen["localhsize"]
  end
        if chars >= max_length or width > localhsize then
I don't think that's necessary. \hsize is a primitive TeX parameter that sets the width of the paragraph. It may be zero at the start of the document, but it is definitely non-zero by the end of every paragraph.
The Lua function gets the current value of \hsize at the end of every paragraph, so it should be using the exact same value that TeX's paragraph builder uses, meaning that it should account for itemizations and such. I'm not really sure what \localhsize is, but it's probably similar to \hsize.
...
(2) I'm (now?) running into trouble with hyphenation.
In my own document, I also get lines with only a single character or hboxed group. I assume, this is because the hyphen is not counted and pushes the remainder to a new line where the intended breakpoint again starts another one.
Try this:
\startluacode
       local max_length = 112
local glyph_id = node.id "glyph"
       local disc_id = node.id "disc"
       local glue_id = node.id "glue"
function userdata.limiter(head)
           language.hyphenate(head)
local hyphen = node.new "glyph"
           hyphen.char = language.prehyphenchar(0)
           hyphen.font = font.current()
           local width = hyphen.width
           node.free(hyphen)
local chars = 0
           local n = head
           while n do
               if n.id == glyph_id or n.id == glue_id then
                   chars = chars + 1
                   width = width + n.width - (n.shrink or 0)
               end
if chars >= max_length or width > tex.hsize then
                   local back_chars = 0
                   local end_disc = nil
while n do
                       if n.id == glue_id then
                           local penalty = node.new "penalty"
                           penalty.penalty = -10000
                           node.insertbefore(head, n, penalty)
                           break
                       end
if not end_disc and n.id == disc_id then
                           end_disc = n
                       end
if end_disc and back_chars >= 5 then
                           end_disc.penalty = -10000
                           break
                       end
if n.id == glyph_id then
                           back_chars = back_chars + 1
                       end
n = n.prev
                   end
width = 0
                   chars = 0
               end
n = n.next
           end
return head
       end
nodes.tasks.appendaction(
           "processors",
           "before",
           "userdata.limiter"
       )
   \stopluacode
I've just added the width of a hyphen to the accumulated width. Let me know if this works; if not, there's a more complex fix that I can try.
...
Unfortunately, I don't know what to change; I know a bit about "glyph" and "glue", but what is "disc" and would it help here?
"disc" nodes are "discretionaries", which are usually potential hyphens. See "The TeXbook" (page 95) or "TeX by Topic" (https://texdoc.org/serve/texbytopic/0#subsection.19.3.1) for details on the TeX side, or the LuaMetaTeX manual (https://www.pragma-ade.com/general/manuals/luametatex.pdf#%231205) for details on the Lua side.
-- Max

Re: [NTG-context] Count (and limit) glyphs per line?

Benjamin Buchmuller