Hi all,

I've pushed lwc v2.1.0 to CTAN and the Garden Modules site. This new version fully supports the ConTeXt grid snapping (MkIV/MkXL), and it also adds some improved logging. You can download this directly at

https://modules.contextgarden.net/dl/lua-widow-control-v2.1.0.zip

I believe that this resolves all of the issues in this thread, but as always, feel free to let me know if something doesn't work.

Thanks, -- Max

Hi, Max:

First things first.  I want to acknowledge and thank you for the tough mission that surely involves maintaining this module for the benefit of the TeX community and, most especially, for LMTX in particular, due to the very reasons you have just explained.

Regarding your remarks, you are right in your assumptions: I didn't have the required lua file installed in its corresponding folder. That's what I was missing, and, logically, what made my compilation crash.

To make my confusion even worse, I had done a strict search inside the mkxl code. It wouldn't take me to line 63 because I had added the extension (lua-widow-control.lua, not just lua-widow-control).

Anyway, a mess only due to my ignorance.

Now I finally get the pdf. But unfortunately, this "new" pdf is the same with the module uncommented as when I had it commented. There is no difference at all. And you're right: even context --make doesn't solve the problem. To give you an idea, although my dummy document has 78 pages, only 55 can really be said to be dummy text that can benefit from the module. Well, only in those 55 pages I have 2 orphans, 5 widows and 6 broken hyphens.

I have read several times all the options of the module in the /lua-widow-control.pdf/ document, but I don't even dare to mess with those options because it seems to me very radical that my output pdf is the same with the module uncommented in my code.

Of course, mtxrun --find-file lua-widow-control.lua yields what you would expect:

/opt/luametatex/texmf-modules/tex/luatex/lua-widow-control/lua-widow-control.lua

Thanks again for your work and yours observations.

Edu.

El 25/4/22 a las 22:00, Max Chernoff escribió:

(Please keep me CC'd as I'm not subscribed to the list)

Hi, I'm the lua-widow-control author.

...

lua error > lua error on line 74 in file

/opt/luametatex/texmf-modules/tex/context/third/lua-widow-control/t-lua-widow-control.mkxl

...

The odd thing is that line 75 of the t-lua-widow-control file is empty.

The \setuplwc command ends on line 74, which then triggers \everysetuplwc which then calls \ctxlua{lwc.enable_callbacks()}. This fails since lwc is undefined because the Lua file isn't loaded because ConTeXt can't seem to find the file.

In Plain LuaTeX and LuaLaTeX, a missing Lua file is a fatal error:

$ luatex "\nonstopmode\directlua{require 'not-a-real-file'}\bye"

This is LuaTeX, Version 1.13.2 (TeX Live 2021/W32TeX)      restricted system commands enabled.

[\directlua]:1: module 'not-a-real-file' not found:             no field package.preload['not-a-real-file']             [kpse lua searcher] file not found: 'not-a-real-file'     stack traceback:             [C]: in function 'require'             [\directlua]:1: in main chunk.     <*> \nonstopmode\directlua{require 'not-a-real-file'}                                                       \bye     (see the transcript file for additional information)     warning  (pdf backend): no pages of output.     Transcript written on texput.log.

so I'm a little surprised that ConTeXt just issues a warning here when it can't find the file:

resolvers > lua > unknown file 'lua-widow-control.lua'

...

when it says it doesn't know the lua-widow-control.lua file. I don't

see

...

any mention of this file within the t-lua-widow-control module.

lua-widow-control.lua is loaded at line 63 of the .mkxl:     \ctxloadluafile{lua-widow-control}

...

Could it be that this module is not yet mature for lmtx

Lua-widow-control is certainly more stable with Plain/LaTeX, but it usually runs fine with LMTX. The entire lwc manual is written in ConTeXt/LMTX so I (usually) notice pretty quickly when things break.

The only real "issue" with LMTX is that the engine changes pretty quickly, so lwc may sometimes be broken for a few days between an engine update and whenever I push out a fix. This doesn't happen with Plain/LaTeX since the LuaTeX engine is mostly frozen.

Interestingly, the MkXL version of lwc actually predates the MkIV version, although only by a few months.

...

I have \usemodule[lua-widow-control] in my tex document, and I haven't forgotten to do the prescribed mtxrun --generate after including the module in my third-party folder of my luametatex installation on Arch.

Hmm, that's odd then. I'm not entirely sure why this is happening, so I'm going to take a random guess:

Maybe you installed lwc using a zipfile from either GitHub or the ConTeXt Garden modules site, then you copied the files into your texmf-modules/ folder, *but* you only copied the "tex/context/" folder and not all of the folders in the "tex" folder. The "lua-widow-control.lua" file is in the "tex/luatex/" folder, so if you didn't also copy that across you're going to have problems.

Again, just a random guess.  If that doesn't work, you could maybe try running     context --make but I doubt that that would fix anything here. You could also try deleting the filename cache files at:     {CONTEXT}/texmf-cache/luametatex-cache/context/*/trees/*.lu? then rerunning     mtxrun --generate I've never deleted those files before, but it's in a folder called "cache" so it's _probably_ safe to delete (make a backup just in case though).

If none of that works, then the issue could be something else on your end, or I could have screwed something up in lwc. In that case, try running     find /opt/luametatex/texmf-modules/ -path '*widow*' -print and     mtxrun --find-file lua-widow-control.lua and reply with the output.

Oh, and one more note. ConTeXt version "2022-04-11 19:20" renamed some of the engine Lua functions, so make sure to update to lwc v2.0.6 (if you're using a recent ConTeXt version). Otherwise after you fix the "unknown file" error, you would get an "attempt to call a nil value" error which isn't any better.

Let me know if you have any other issues.

-- Max

Hi:

No line such as "Widow/orphan detected. Attempting to delete".

I see interleaved new groups with the same line always repeating a warning message throughout the whole file. In short, there are 613 new lines with the message "luatex warning > tex: left parfill skip is gone". But I didn't give it any importance, because I interpreted that they could be inherent to the module.

I can see "modules         > 'lua-widow-control' is loaded".

But, luckily, I can also see this:

open source     > level 2, order 4, name '/opt/luametatex/texmf-modules/tex/context/third/lua-widow-control/t-lua-widow-control.mkxl' resolvers       > lua > loading file '/opt/luametatex/texmf-modules/tex/luatex/lua-widow-control/lua-widow-control.lua' succeeded close source    > level 2, order 4, name '/opt/luametatex/texmf-modules/tex/context/third/lua-widow-control/t-lua-widow-control.mkxl' module          > lua-widow-control > Already enabled

On the other hand, the distribution and size of my paragraphs take great care that their "design" optimises the module's goodness, except, of course, for the first pages of each of the nine dummy text chapters (they start at a third of a page). But it is very curious that, even so, only the two orphan lines I mentioned, and only two of the five widows relate to two or three of those "supposedly problematic" first chapter pages (as I said, nine in total). And the case of the broken scripts is even stranger: only two of the six breakages concern a couple of those early chapter pages.

I will go over the lua-widow-control.pdf document once more in case there is a tiny detail I am missing, but I think, if I keep going at this pace, I will end up learning it by heart.

Well, joking aside, thanks again for your advice, Max.

Edu.

El 26/4/22 a las 4:42, Max Chernoff escribió:

On 2022-04-25 6:51 p.m., Eduardo Bohoyo wrote:

...

First things first.  I want to acknowledge and thank you for the tough mission that surely involves maintaining this module for the benefit of the TeX community and, most especially, for LMTX in particular, due to the very reasons you have just explained.

Well thanks :) These days I write most of my documents in LMTX, so the LMTX support is pretty self-serving -- I'm admittedly surprised that there's another lwc + LMTX user.

...

Regarding your remarks, you are right in your assumptions: I didn't have the required lua file installed in its corresponding folder. That's what I was missing, and, logically, what made my compilation crash.

Yeah, it's a pretty easy mistake to make. For Knuth TeX, shared files go in texmf/tex/generic and Plain-specific files go in texmf/tex/plain, but with LuaTeX there's just texmf/tex/luatex so it gets a little confusing.

...

Now I finally get the pdf. But unfortunately, this "new" pdf is the same with the module uncommented as when I had it commented. There is no difference at all. And you're right: even context --make doesn't solve the problem. To give you an idea, although my dummy document has 78 pages, only 55 can really be said to be dummy text that can benefit from the module. Well, only in those 55 pages I have 2 orphans, 5 widows and 6 broken hyphens.

So the first step here is to check the log file. If you see lines like

module          > lua-widow-control > Widow/orphan detected. Attempting to remove.     module          > lua-widow-control > Widow/Orphan NOT removed on page X.

then that means that lwc found a widow/orphan, but gave up. This usually only happens if the page has only really short paragraphs, but it can also happen if there aren't any paragraphs that both start and finish on the page. I've got some neat graphs for this (see the upcoming TUGboat issue), but with default settings this should happen for much less than 10% of potential widows/orphans, so it seems unlikely that this is happening for every page.

If this actually is the issue, then you can try raising the "emergencystretch" value in "\setuplwc", but that's probably going to give terrible results. The real solution is to rewrite something, but that should usually be pretty rare.

---

If you see lines like

module          > lua-widow-control > Widow/orphan detected. Attempting to remove.     module          > lua-widow-control > Widow/orphan successfully removed at paragraph X on page Y.

but the widows/orphans weren't actually removed, then something really weird is going on and definitely means that there's a bug in lwc. Rerun the document with

\setuplwc[debug=true]

immediately after "\usemodule[lua-widow-control]" and either reply with the log or post a new issue on the lwc GitHub.

---

If you only see

modules         > 'lua-widow-control' is loaded

but not any other lines mentioning lwc, then something weird is going on, and lwc is loaded but not enabled. If this is the case, same as last time: enable debug mode and reply with the log.

---

If you don't see

modules         > 'lua-widow-control' is loaded

at all, then you're not actually loading lwc. This probably means that ConTeXt can't find the file, there should hopefully be clues somewhere in the log file.

...

Thanks again for your work and yours observations.

No problem; glad to help.

-- Max

Quick question before I begin: are you using any especially "interesting" ConTeXt features? By "interesting" I mean things like grid typesetting, pagecolumns, bidirectional text, etc. I haven't tested lwc with every possible ConTeXt feature, so there may be some adverse interaction. If you are using something like this, try disabling it and see if that solves anything (then let me know so that I can fix it!)

On 2022-04-26 3:45 a.m., Eduardo Bohoyo wrote:

I can see "modules         > 'lua-widow-control' is loaded".

But, luckily, I can also see this:

open source     > level 2, order 4, name '/opt/luametatex/texmf-modules/tex/context/third/lua-widow-control/t-lua-widow-control.mkxl'

resolvers       > lua > loading file '/opt/luametatex/texmf-modules/tex/luatex/lua-widow-control/lua-widow-control.lua' succeeded close source    > level 2, order 4, name '/opt/luametatex/texmf-modules/tex/context/third/lua-widow-control/t-lua-widow-control.mkxl'

module          > lua-widow-control > Already enabled

Ok, so this is good; lwc is for sure being loaded successfully.

No line such as "Widow/orphan detected. Attempting to delete".

I see interleaved new groups with the same line always repeating a warning message throughout the whole file. In short, there are 613 new lines with the message "luatex warning > tex: left parfill skip is gone". But I didn't give it any importance, because I interpreted that they could be inherent to the module.

Well this at least narrows the issue down quite a bit. Lwc runs in pretty much two stages: when a paragraph has finished being broken by TeX, lwc saves the paragraph. The second stage is ran just before each time the output routine is triggered so that lwc can remove the widows and orphans.

Due to an lwc bug, the first stage results in the "left parfill skip" warning being printed twice for each paragraph. Normally this is quite annoying, but here it is good -- we know for sure that the first stage is running just fine.

It is the second stage where you should get the "Widow/orphan detected" message, but this isn't happening. The code here is at lwc.lua:362-388.

Here is a list of all possible reasons, in order of likelihood, why "Widow/orphan detected" wouldn't be printed when there is actually a widow or orphan:

(Just listing all of these to make sure that *I* don't forget anything. I'd say that 1 and 2 are the only ones that are actually likely -- you can probably ignore all of the others)

1. "\clubpenalty" and/or "\widowpenalty" are either zero or infinite

In these cases, lwc can't distinguish a widow or orphan from other penalties, so it doesn't do anything. You can check for this with

\showthe\widowpenalty \showthe\clubpenalty

some time after "\starttext". This will print the values to the log file. The values should probably be "1", but things like "150" and "5000" are fine too. If you see negative values, "0", or values greater than ten thousand, then something is wrong here.

2. Something is "hiding" the widow/orphan penalties from lwc.

Something like "\vadjust{\penalty X}" or the e-TeX "-penalties" commands could potentially do this.

One potential solution for (a subset of) this problem would be to modify lwc.mkxl to insert "\directsetup{*reset}" at line 52, to get:

\startsetups[*default] \directsetup{*reset} \clubpenalty=\lwcparameter{orphanpenalty} \widowpenalty=\lwcparameter{widowpenalty} \displaywidowpenalty=\lwcparameter{widowpenalty} \brokenpenalty=\lwcparameter{brokenpenalty} \stopsetups

If that doesn't do anything, keep the above modifications but also insert

\setups[*default]

after "\starttext" in your test document.

3. Lwc hasn't saved any paragraphs since the last output routine.

Since we know that the first stage is running, this shouldn't be possible unless there is a weird bug somewhere else.

4. A weird engine/format bug.

The LMTX test file was successful on Debian x86_64 three days ago

https://github.com/gucci-on-fleek/lua-widow-control/runs/6144354147?check_su...

so something quite weird would have to be going on here.

5. Something is overwriting the "pre_output_filter" callback.

Now this is an interesting one. ConTeXt has frozen most of the core engine callbacks, but "pre_output_filter" is unfrozen and unassigned. There is a ConTeXt interface for most of the other callbacks, but here I need to fallback to the low-level interface. With this interface, you can only register a single callback, so it is possible that some other package is overwriting the callback. To test this, the very beginning of your document should look like:

\usemodule[lua-widow-control] \startluacode lwc.callbacks.disable_box_warnings.enable = function () end lwc.callbacks.disable_box_warnings.disable = function () end callback = nil \stopluacode

If you get a really strange error from some other package/module (not lwc), then this may be the issue. Fairly unlikely.

6. The values of "\club/widowpenalty" change between the paragraph being typeset and the output routine being triggered.

I can't imagine this happening for *every* page in a document except unless you are specifically trying to do this. Fairly unlikely.

7. You have the exact Lua code

lwc.callbacks.remove_widows.disable()

written in your document. Exceedingly unlikely.

On the other hand, the distribution and size of my paragraphs take great care that their "design" optimises the module's goodness, except, of course, for the first pages of each of the nine dummy text chapters (they start at a third of a page).

Well as long as you aren't seeing

Widow/Orphan NOT removed

in your log file then your text should be fine.

I will go over the lua-widow-control.pdf document once more in case there is a tiny detail I am missing, but I think, if I keep going at this pace, I will end up learning it by heart.

There's probably not too much else in there -- and the documentation is a bit of a mess right now anyways.

---

If none of the above solutions solved anything, then what I'll need to debug further will be for you to insert

\setuplwc[debug=true]

into your document right after you load lwc. Then, send me the full log file -- probably as a link since it will be quite large.

Thanks.

-- Max