------ Original Message ------
From "Hans Hagen via ntg-context" mailto:ntg-context@ntg.nl>
To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl>
Cc "Hans Hagen" mailto:j.hagen@freedom.nl>
Date 9/22/2023 10:44:38 AM
Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters
(It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.)
I admit that I don't follow what happens with xetex (they changed the
rendere at some point indeed) not HB (I only notice that it gets updates
frequently in the tex live repository which makes me wonder how one
retains compatility unless one freezes). I actually kept the lib binding
code that can use it around for your font testing (we wanted to see what
uniscribe does), not sure if it still works.
I remember the experimental uniscribe binding, don't recall that it was ever officially released. (Did Kai Bormann ever work with it?).
I hope that it still works or can be resurrected when the time comes for testing -)
Anyway, we're entering the bug cq. side effect becomes feature area
here; just like yesterdays perfect bidi algorithm is todays less pefect
one replaced by ...
But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.
if context shows it then it is not a feature but hard coded shapes which
is weird; how does one know what to 'remove' or not? And in what stage?
If they are zero width it is simple to ignore them in the backend, if
they have dimensions (w/h/d) then they contributed and wiping is tricky
The metrics should be dimensionless and most Arabic fonts do that, but Lotus Linotype gives them width (one wonders why).
Perhaps the Lotus designers thought that giving them width makes them easier to edit, which is indeed the case. And perhaps they were satisfied that Word/InDesign would be able, as needed, to toggle the metrics on and off as well as the shape. That WYSIWYG-DTP curse again..
\setcharacterstripping[1] gives the right results with Lotus (at least so far) -- see attached. Was pleasantly surprised that the spurious width was also suppressed.
\setcharacterstripping[0] toggles back -- another pleasant result -)
Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted.
(I suppose that InDesign behaves the same way.)
I don't know ... irr these dtp programs are more like "if you want this
feature applied select a range of characters and apply it"
That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue.
In some way it's also flaws in the open type approach. Basically that
happens when application stuff becomes a standard and one forgets that
it was (is) application driven. (And you haven't seen variable fonts and
color fonts yet ... no pretty standards either.)
I'm in the process of switching from Multiple Masters sources to variable, so will surely encounter it soon.
Unfortunately there is indeed a frustrating gray area in this arena between knowing where the standards end and where the diktats of corporations (or even free-software consortiums like HB) begin, as we've discussed in the past. With variable fonts, one has to add some degree of arbitrariness..
Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.
Funny rules ... but I'm not going top enable 'wipe' by default: after
all, one gets what one deserves, nto what one likes (which can differ
per day). But you can enable the wiping. We can of course ignore in the
backend when zero width but then how to explain that they contributed to
the ht/dp (unless we wipe these dimensions) ... all slow-downers
So far \setcharacterstripping[1] is giving the right results. If the relevant trackers are available, could add them to the attached test. But presumably you know how and why it works even for a non-dimensionless glyph.
so you want to see soem zwj sumbol in a rendered text?
Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -)
Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does.
(Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!)
We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.)
so what is it now:
- for verbatim you can use almfixed and they show up (when they have a
glyph)
Except for \nobreakspace I think. At some point it may be worth seeing if there is a way to get the NBSP glyph to show in a typed buffer, or if the color-representation can be configured as some symbol instead of a color. But not urgent.
- for other fonts if they have them they show up (unless gone in the
process ot rendering)
- but you can wipe them optionally
not sure what more we need
For now I think we're good. If anything comes up during the immediate project, will post it.
Many thanks as always Hans.
Best wishes
--
Idris Samawi Hamid, Professor
Department of Philosophy
Colorado State University
Fort Collins, CO 80523
