Toggling the symbol for the zero-width joiner and related Unicode control characters

Hamid,Idris

21 Sep 2023 21 Sep '23

7:31 a.m.

Dear gang, Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained. In some text editors, such as Windows Notepad (Uniscribe), there is the option "Show Unicode control characters" which toggles the symbols and keeps the effects. I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it. Does anyone know or recall where the symbol-toggling mechanism is? If not, what would be the simplest way to implement it? Here is a MWE (note that the three control chars might be invisible in your email client): ============== \startTEXpage[offset=1em] \definedfont[almfixed at 14pt] ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage ============== In Notepad, all three control characters can be turned on or off at once. In Scintilla, only the NBSP appears and there is no option to toggle. In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not. Thank you in advance for your guidance. Best wishes Idris -- Idris Samawi Hamid, Professor Department of Philosophy Colorado State University Fort Collins, CO 80523

Attachments:

attachment.html (text/html — 2.8 KB)

Show replies by date

Hans Hagen

21 Sep 21 Sep

8:52 a.m.

On 9/21/2023 3:31 PM, Hamid,Idris wrote:

...

Dear gang,

Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained.

In some text editors, such as Windows Notepad (Uniscribe), there is the option

"Show Unicode control characters"

which toggles the symbols and keeps the effects.

I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it.

Does anyone know or recall where the symbol-toggling mechanism is? If not, what would be the simplest way to implement it?

Here is a MWE (note that the three control chars might be invisible in your email client):

============== \startTEXpage[offset=1em] \definedfont[almfixed at 14pt] ZWJ: ‌

ZWNJ: ‍

NBSP: \stopTEXpage ==============

In Notepad, all three control characters can be turned on or off at once. In Scintilla, only the NBSP appears and there is no option to toggle. In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not.

Thank you in advance for your guidance. \enabletrackers[typesetters.nbsp]

----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hamid,Idris

2:53 p.m.

------ Original Message ------ From "Hans Hagen" mailto:j.hagen@xs4all.nl> To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl> Date 9/21/2023 8:52:45 AM Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters ** Caution: EXTERNAL Sender ** On 9/21/2023 3:31 PM, Hamid,Idris wrote: Dear gang, Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained. In some text editors, such as Windows Notepad (Uniscribe), there is the option "Show Unicode control characters" which toggles the symbols and keeps the effects. I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it. Does anyone know or recall where the symbol-toggling mechanism is? If not, what would be the simplest way to implement it? Here is a MWE (note that the three control chars might be invisible in your email client): ============== \startTEXpage[offset=1em] \definedfont[almfixed at 14pt] ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage ============== In Notepad, all three control characters can be turned on or off at once. In Scintilla, only the NBSP appears and there is no option to toggle. In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not. Thank you in advance for your guidance. \enabletrackers[typesetters.nbsp] Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE: ======= \startTEXpage[offset=1em] \disabletrackers[typesetters.directions] \disabletrackers[typesetters.zwj] \disabletrackers[typesetters.zwnj] \disabletrackers[typesetters.nbsp] \definedfont[almfixed at 14pt] ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage ======= See attached, please advise. Best wishes Idris -- Idris Samawi Hamid, Professor Department of Philosophy Colorado State University Fort Collins, CO 80523

Hans Hagen

3:29 p.m.

On 9/21/2023 10:53 PM, Hamid,Idris wrote:

...

------ Original Message ------ From "Hans Hagen" mailto:j.hagen@xs4all.nl> To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl> Date 9/21/2023 8:52:45 AM Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

** Caution: EXTERNAL Sender **

On 9/21/2023 3:31 PM, Hamid,Idris wrote: Dear gang,

Within a font, relevant Unicode control characters such as the zero-width joiner usually have symbolic representation, but in printing the symbols are usually suppressed, while the effect of each character is maintained.

In some text editors, such as Windows Notepad (Uniscribe), there is the option

"Show Unicode control characters"

which toggles the symbols and keeps the effects.

I seem to recall that ConTeXt has/had a similar mechanism for toggling the symbols for Unicode control characters but cannot find it.

Does anyone know or recall where the symbol-toggling mechanism is? If not, what would be the simplest way to implement it?

Here is a MWE (note that the three control chars might be invisible in your email client):

============== \startTEXpage[offset=1em] \definedfont[almfixed at 14pt] ZWJ: ‌

ZWNJ: ‍

NBSP: \stopTEXpage ==============

In Notepad, all three control characters can be turned on or off at once. In Scintilla, only the NBSP appears and there is no option to toggle. In ConTeXt, ZWJ and ZWJ appear as symbols, but NBSP does not.

Thank you in advance for your guidance. \enabletrackers[typesetters.nbsp]

Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE:

======= \startTEXpage[offset=1em] \disabletrackers[typesetters.directions] \disabletrackers[typesetters.zwj] \disabletrackers[typesetters.zwnj] \disabletrackers[typesetters.nbsp] \definedfont[almfixed at 14pt] ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage =======

See attached, please advise.

joiners are part of replacement etc and can come and go ... they are characters (we could visualize them but one never knows for sure if one sees them) nbsp are spaces and become glue that we can trace reliable in the node list Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hamid,Idris

10:16 p.m.

------ Original Message ------ From "Hans Hagen" mailto:j.hagen@xs4all.nl> To "Hamid,Idris" mailto:Idris.Hamid@ColoState.EDU>; "mailing list for ConTeXt users" mailto:ntg-context@ntg.nl> Date 9/21/2023 3:29:22 PM Subject Re: [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE: ======= \startTEXpage[offset=1em] \disabletrackers[typesetters.directions] \disabletrackers[typesetters.zwj] \disabletrackers[typesetters.zwnj] \disabletrackers[typesetters.nbsp] \definedfont[almfixed at 14pt] ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage ======= See attached, please advise. joiners are part of replacement etc and can come and go ... they are characters (we could visualize them but one never knows for sure if one sees them) nbsp are spaces and become glue that we can trace reliable in the node list Many thanks. Ok, here is another MWE featuring a workaround using fallbacks: ============== \definefontfallback[nosymbols] [file:lmmono10-regular] [200C,200D] [force=yes] \starttypescript [serif] [alm] [name] \definefontsynonym [Serif] [ArabicLatinSerif] \stoptypescript \starttypescript [mono] [alm] [name] \definefontsynonym [Mono] [ArabicLatinMono] \stoptypescript \starttypescript [serif] [alm] \definefontsynonym [ArabicLatinSerif] [file:almfixed] % [fallbacks=nosymbols] \stoptypescript \starttypescript [mono] [alm] \definefontsynonym [ArabicLatinMono] [file:almfixed] [fallbacks=nosymbols] \stoptypescript \starttypescript [almfixed-nosymbols] \definetypeface [\typescriptone] [rm] [serif] [alm] [default] \definetypeface [\typescriptone] [tt] [mono] [alm] [default] \stoptypescript \usetypescript[almfixed-nosymbols] \setupbodyfont[almfixed-nosymbols,12pt] \startTEXpage[offset=1em] \rm ZWJ: ‌ ZWNJ: ‍ NBSP: \tt ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage ============== Under \rm we get the symbols, and under \tt they are suppressed. Of course it doesn't matter what fallback font one uses, as long as it has no control-character symbols. 1. Can this approach be generalized to get what we want, viz., a way to toggle the symbols? 2. \enabletrackers[typesetters.nbsp] gives a colored box, which is at least something.. But how can we get the NBSP symbol that's alerady in the font? 3. Ideally: a. we want all Unicode control symbols to show up in verbatim or in \typebuffer (as in a text editor); b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing). But some fonts meant for printing have symbols for Unicode control chars -- that poses a challenge. And some fonts meant for verbatim/editing do not have symbols for the control chars -- that also poses a challenge. AlmFixed, of course, has them. Most minimally decent Arabic fonts have symbols for the Unicode control chars as default, including Scheherazade, Amiri, Uthmanic, and Noto Naskh Arabic -- all free fonts. Industry workhorses like Linotype Lotus (Arabic) also have them. Uniscribe applications like Notepad/Word allow for toggling in a WYSIWYG context -- can't speak for HarfBuzz -- so there is no harm in having explicit symbols in the font. The upshot is that, for non-Latin scripts, some toggling capability in ConTeXt is important to have -- even inescapable for Arabic-script piblishing. Perhaps others who use Arabic-script or Indic, etc., can chime in.. Am hopeful that we can figure something out! Best wishes Idris -- Idris Samawi Hamid, Professor Department of Philosophy Colorado State University Fort Collins, CO 80523

Hans Hagen

22 Sep 22 Sep

3:53 a.m.

On 9/22/2023 6:16 AM, Hamid,Idris wrote:

...

------ Original Message ------ From "Hans Hagen" mailto:j.hagen@xs4all.nl> To "Hamid,Idris" mailto:Idris.Hamid@ColoState.EDU>; "mailing list for ConTeXt users" mailto:ntg-context@ntg.nl> Date 9/21/2023 3:29:22 PM Subject Re: [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters Many thanks, Hans. The method appears to work only for nbsp, not zwj etc. Here is the updated MWE:

======= \startTEXpage[offset=1em] \disabletrackers[typesetters.directions] \disabletrackers[typesetters.zwj] \disabletrackers[typesetters.zwnj] \disabletrackers[typesetters.nbsp] \definedfont[almfixed at 14pt] ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage =======

See attached, please advise. joiners are part of replacement etc and can come and go ... they are characters (we could visualize them but one never knows for sure if one sees them)

nbsp are spaces and become glue that we can trace reliable in the node list

Many thanks. Ok, here is another MWE featuring a workaround using fallbacks:

============== \definefontfallback[nosymbols] [file:lmmono10-regular] [200C,200D] [force=yes] \starttypescript [serif] [alm] [name] \definefontsynonym [Serif] [ArabicLatinSerif] \stoptypescript \starttypescript [mono] [alm] [name] \definefontsynonym [Mono] [ArabicLatinMono] \stoptypescript \starttypescript [serif] [alm] \definefontsynonym [ArabicLatinSerif] [file:almfixed] % [fallbacks=nosymbols] \stoptypescript \starttypescript [mono] [alm] \definefontsynonym [ArabicLatinMono] [file:almfixed] [fallbacks=nosymbols] \stoptypescript \starttypescript [almfixed-nosymbols] \definetypeface [\typescriptone] [rm] [serif] [alm] [default] \definetypeface [\typescriptone] [tt] [mono] [alm] [default] \stoptypescript \usetypescript[almfixed-nosymbols] \setupbodyfont[almfixed-nosymbols,12pt] \startTEXpage[offset=1em] \rm ZWJ: ‌ ZWNJ: ‍ NBSP: \tt ZWJ: ‌ ZWNJ: ‍ NBSP: \stopTEXpage ==============

Under \rm we get the symbols, and under \tt they are suppressed. Of course it doesn't matter what fallback font one uses, as long as it has no control-character symbols.

1. Can this approach be generalized to get what we want, viz., a way to toggle the symbols?

given the inconsistency in what is or is not in a font the only way out is to have our own visualization (consistent across fonts) and even then it would add some mess because we're talking of a mix of characters that can have gone (as part of rendering) or are not characters at all but spacing so, in that case only 'verbatim' is a candidate for visualization, not so much typeset text

...

2. \enabletrackers[typesetters.nbsp] gives a colored box, which is at least something.. But how can we get the NBSP symbol that's alerady in the font?

it's gone by that time ... the line break mmechanism uses glue, not characters

...

3. Ideally: a. we want all Unicode control symbols to show up in verbatim or in \typebuffer (as in a text editor);

only there (with some non interfering rendering i guess) and even then it's probably an additonal pass over the node list

...

b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

they basically are unless some font features keeps them around which is out of our control

...

But some fonts meant for printing have symbols for Unicode control chars -- that poses a challenge.

so an inconsistent mess not worth wasting time on (as this is hobbyism only fun can be a motivational factir)

...

And some fonts meant for verbatim/editing do not have symbols for the control chars -- that also poses a challenge. AlmFixed, of course, has them.

Most minimally decent Arabic fonts have symbols for the Unicode control chars as default, including Scheherazade, Amiri, Uthmanic, and Noto Naskh Arabic -- all free fonts.

Industry workhorses like Linotype Lotus (Arabic) also have them.

i'm not interested in those .. can't afford them for playing around purposes .. we only look into commercial fonts if we get a dozen unresticted copies for context developers

...

Uniscribe applications like Notepad/Word allow for toggling in a WYSIWYG context -- can't speak for HarfBuzz -- so there is no harm in having explicit symbols in the font.

sure, as long as there is no rendering ... they show the input

...

The upshot is that, for non-Latin scripts, some toggling capability in ConTeXt is important to have -- even inescapable for Arabic-script piblishing.

a bit subjectiev arguing -)

...

Perhaps others who use Arabic-script or Indic, etc., can chime in.. Am hopeful that we can figure something out! sore, but not with 'instant priority' (unless it is some project)

Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hamid,Idris

6:39 a.m.

------ Original Message ------ From "Hans Hagen via ntg-context" mailto:ntg-context@ntg.nl> To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl> Cc "Hans Hagen" mailto:j.hagen@freedom.nl> Date 9/22/2023 3:53:03 AM Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters 1. Can this approach be generalized to get what we want, viz., a way to toggle the symbols? given the inconsistency in what is or is not in a font the only way out is to have our own visualization (consistent across fonts) and even then it would add some mess because we're talking of a mix of characters that can have gone (as part of rendering) or are not characters at all but spacing so, in that case only 'verbatim' is a candidate for visualization, not so much typeset text Hm, ok. Since almfixed is based on Knuth's mono, perhaps its visuals of the control characters can be extracted and used as fallback symbols. Yes: For typeset text/printing visualization is generally unnecessary (the point of this thread). 2. \enabletrackers[typesetters.nbsp] gives a colored box, which is at least something.. But how can we get the NBSP symbol that's alerady in the font? it's gone by that time ... the line break mmechanism uses glue, not characters Ok 3. Ideally: a. we want all Unicode control symbols to show up in verbatim or in \typebuffer (as in a text editor); only there (with some non interfering rendering i guess) and even then it's probably an additonal pass over the node list Ok, that would be good. b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing). they basically are unless some font features keeps them around which is out of our control If the symbols are in the font, then they are not suppressed. See below. But some fonts meant for printing have symbols for Unicode control chars -- that poses a challenge. so an inconsistent mess not worth wasting time on (as this is hobbyism only fun can be a motivational factir) But there is a certain consistency -- see below. And some fonts meant for verbatim/editing do not have symbols for the control chars -- that also poses a challenge. AlmFixed, of course, has them. Most minimally decent Arabic fonts have symbols for the Unicode control chars as default, including Scheherazade, Amiri, Uthmanic, and Noto Naskh Arabic -- all free fonts. Industry workhorses like Linotype Lotus (Arabic) also have them. i'm not interested in those .. can't afford them for playing around purposes .. we only look into commercial fonts if we get a dozen unresticted copies for context developers Except for Linotype Lotus, each of the Arabic-script fonts mentioned above is free, not commercial -) (There is also a free version of Lotus -- it also has the symbolic rendering of the contol chars.) Uniscribe applications like Notepad/Word allow for toggling in a WYSIWYG context -- can't speak for HarfBuzz -- so there is no harm in having explicit symbols in the font. sure, as long as there is no rendering ... they show the input But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue. Yet another curse of the WYSIWYG paradigm, which mixes form and content -) The upshot is that, for non-Latin scripts, some toggling capability in ConTeXt is important to have -- even inescapable for Arabic-script piblishing. a bit subjectiev arguing -) Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned. (Yes, for the upcoming Husayni I can add a font feature that does the trick, but that will be an exception to the rule.) Perhaps others who use Arabic-script or Indic, etc., can chime in.. Am hopeful that we can figure something out! sore, but not with 'instant priority' (unless it is some project) My immediate project (no Husayni) is a book that features English translation of an Arabic text (hence the interest in the recent streams thread). Using some Unicode control characters will be unavoidable to get the rendering effects correct, but the symbols will need to be suppressed. Am thinking/hoping that a ConTeXt-specific font feature can do the trick. Since there appears to be consistency across Arabic fonts in this matter it should not be messy at all, simply a fallback that sends the symbols to some no-man's land. (A thought: Some of the code you kindly provided for transliteration might be reusable as well.. But a general solution for all ConTeXt users would be ideal.) In any case, many thanks for your help in thinking this through. Best wishes Idris -- Idris Samawi Hamid, Professor Department of Philosophy Colorado State University Fort Collins, CO 80523

Hans Hagen

7:03 a.m.

Hi, I found it ... \startbuffer \definedfont[almfixed*default]hello w\zwnj o\zwj r\zwnj l\zwj d \stopbuffer \getbuffer \start \setcharacterstripping[1] \getbuffer \stop so now, being its only user, you have to wikify it ... Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hamid,Idris

7:22 a.m.

Hans Hagen

7:15 a.m.

On 9/22/2023 2:39 PM, Hamid,Idris wrote:

...

b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

they basically are unless some font features keeps them around which is out of our control

irr it was you who wanted them to be wiped decades ago as some fonts visualized them by default

...

But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

...

Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

so you want to see soem zwj sumbol in a rendered text? Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Hamid,Idris

7:51 a.m.

------ Original Message ------ From "Hans Hagen" mailto:j.hagen@xs4all.nl> To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl> Date 9/22/2023 7:15:25 AM Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters ** Caution: EXTERNAL Sender ** On 9/22/2023 2:39 PM, Hamid,Idris wrote: b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing). they basically are unless some font features keeps them around which is out of our control irr it was you who wanted them to be wiped decades ago as some fonts visualized them by default Yes, that's exactly the point: Somewhere along the course of history, it became standard for Arabic-script fonts (and other cursive-script fonts as well) to include symbols for the control characters. In typo-rep there is also %D \starttyping %D \definefontfeature[default][default][mode=node,formatters=strip] %D \stoptyping You included some notes about Khaled, so I guess he faced the same issue. His Amiri font displays the symbols by default, as do other Arabic fonts. (It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.) But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue. ? Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted. (I suppose that InDesign behaves the same way.) That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue. Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned. so you want to see soem zwj sumbol in a rendered text? Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -) Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does. (Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!) We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.) I hope the above made sense. Best wishes -- Idris Samawi Hamid, Professor Department of Philosophy Colorado State University Fort Collins, CO 80523

Hans Hagen

10:44 a.m.

On 9/22/2023 3:51 PM, Hamid,Idris wrote:

...

------ Original Message ------ From "Hans Hagen" mailto:j.hagen@xs4all.nl> To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl> Date 9/22/2023 7:15:25 AM Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters

** Caution: EXTERNAL Sender **

On 9/22/2023 2:39 PM, Hamid,Idris wrote:

b. we want all Unicode control symbols to be suppressed in final pdf output (for, e.g., printing).

they basically are unless some font features keeps them around which is out of our control

irr it was you who wanted them to be wiped decades ago as some fonts visualized them by default

Yes, that's exactly the point: Somewhere along the course of history, it became standard for Arabic-script fonts (and other cursive-script fonts as well) to include symbols for the control characters.

In typo-rep there is also

%D \starttyping %D \definefontfeature[default][default][mode=node,formatters=strip] %D \stoptyping

You included some notes about Khaled, so I guess he faced the same issue. His Amiri font displays the symbols by default, as do other Arabic fonts.

(It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.)

I admit that I don't follow what happens with xetex (they changed the rendere at some point indeed) not HB (I only notice that it gets updates frequently in the tex live repository which makes me wonder how one retains compatility unless one freezes). I actually kept the lib binding code that can use it around for your font testing (we wanted to see what uniscribe does), not sure if it still works. Anyway, we're entering the bug cq. side effect becomes feature area here; just like yesterdays perfect bidi algorithm is todays less pefect one replaced by ...

...

But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue.

if context shows it then it is not a feature but hard coded shapes which is weird; how does one know what to 'remove' or not? And in what stage? If they are zero width it is simple to ignore them in the backend, if they have dimensions (w/h/d) then they contributed and wiping is tricky

...

Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted.

(I suppose that InDesign behaves the same way.)

I don't know ... irr these dtp programs are more like "if you want this feature applied select a range of characters and apply it"

...

That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue.

In some way it's also flaws in the open type approach. Basically that happens when application stuff becomes a standard and one forgets that it was (is) application driven. (And you haven't seen variable fonts and color fonts yet ... no pretty standards either.)

...

Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned.

Funny rules ... but I'm not going top enable 'wipe' by default: after all, one gets what one deserves, nto what one likes (which can differ per day). But you can enable the wiping. We can of course ignore in the backend when zero width but then how to explain that they contributed to the ht/dp (unless we wipe these dimensions) ... all slow-downers

...

so you want to see soem zwj sumbol in a rendered text?

Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -)

Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does.

(Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!)

We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.) so what is it now:

- for verbatim you can use almfixed and they show up (when they have a glyph) - for other fonts if they have them they show up (unless gone in the process ot rendering) - but you can wipe them optionally not sure what more we need Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl -----------------------------------------------------------------

Jean-Pierre Delange

23 Sep 23 Sep

6:40 a.m.

New subject: Install fonts on MS Windows

Dear list, I have a very simple question, perhaps too simple... I've installed CTX on MSWindows x64, but I can't use the fonts I've installed, despite the command: set OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts. I cleared the Windows font cache, restarted the machine, then reloaded with the command : mtxrun --script fonts --reload A search for the fonts I had installed (theano didot and gfs didot) gave no results with : mtxrun --script fonts --list --pattern=*gfs* --all. I admit that I don't know in which directory it is best to install the fonts for optimum performance. Any ideas? -- Jean-Pierre Delange Ancients&Moderns "Few discoveries are more irritating than those which expose the pedigree of ideas". Lord Acton

Pablo Rodriguez

11:15 a.m.

New subject: Install fonts on MS Windows

On 9/23/23 14:40, Jean-Pierre Delange wrote:

...

Dear list, I have a very simple question, perhaps too simple... I've installed CTX on MSWindows x64, but I can't use the fonts I've installed, despite the command: set OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

Dear Jean-Pierre, only if you don’t have no way to install fonts on the OS directory for fonts (%WINDIR%\Fonts in Windows), I would use (in that order): 1. %USERPROFILE%\texmf\texmf-fonts\, or as last option 2. [context-dir]\tex\texmf-local\texmf-fonts\ Excuse me if this is obvious to you, %USERPROFILE% is a Windows variable (Windows variables, once defined, are invoked enclosed in percent signs [such as in %variable%] and they are case-insensitive). %USERPROFILE% in Windows is $HOME in Unix. [context-dir] is no variable. It means the path were you have your ConTeXt distribution installed (in case you have a ConTeXt only distribution, and not TeX Live installed [I mean, a full text distribution). BTW, do you really have a directory C:\Windows\Users\adeimantos? Sorry, but I think it reads C:\Users\adeimantos (actually, this would be the value of the Windows variable %USERPROFILE% when you are logged in). [More on Windows variables, https://ss64.com/nt/syntax-variables.html.]

...

I cleared the Windows font cache, restarted the machine, then reloaded with the command : mtxrun --script fonts --reload

I think it is better the following: mtxrun --script cache --erase && mtxrun --generate It erases the whole cache, which will be generated in your next ConTeXt run.

...

A search for the fonts I had installed (theano didot and gfs didot) gave no results with :

mtxrun --script fonts --list --pattern=*gfs* --all.

I think this would do the trick. Compile the following source: \mainlanguage[agr] \definefontfamily[mainface][rm][GFS Didot] \definefontfamily[mainface][ss][Theano Didot] \setupbodyfont[mainface] \starttext \doloopoverlist{\tf, \it, \bf, \bi} {\recursestring\input aristotle-grc\par} \ss\input aristotle-grc \stoptext So you will see whether ConTeXt is able to read the fonts or not (if they are embedded in the output PDF document or not).

...

I admit that I don't know in which directory it is best to install the fonts for optimum performance.

If you install the fonts in %WINDIR%\Fonts (Windows font directory), the will be available for any program. In some cases, this is important, even when using only with ConTeXt. If you need only in TeX, %USERPROFILE%\texmf\texmf-fonts\ would be a good option. But even if you choose the third option above (numbered as two), you don’t need to set the OSFONTDIR variable. On Windows and macOS, the OS font directory is set by ConTeXt itself. The other two directories are also defined. BTW, where have you copied both fonts? Just in case it might help, Pablo

Henning Hraban Ramm

12:05 p.m.

New subject: Install fonts on MS Windows

Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:

...

Dear list, I have a very simple question, perhaps too simple... I've installed CTX on MSWindows x64, but I can't use the fonts I've installed, despite the command: set OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

That should be ok.

...

I cleared the Windows font cache, restarted the machine, then reloaded with the command : mtxrun --script fonts --reload

That should be enough. Maybe try mtxrun --script font --reload --force I need that regularly for cambria.ttc to work (on OSX).

...

A search for the fonts I had installed (theano didot and gfs didot) gave no results with :

mtxrun --script fonts --list --pattern=*gfs* --all.

The pattern doesn’t need jokers. Did you try without?

...

I admit that I don't know in which directory it is best to install the fonts for optimum performance.

If you need the fonts also for other programs, the OS directory is good, otherwise <ConTeXt installation directory>/texmf-fonts/fonts/data/ is recommended. Hraban

Wolfgang Schuster

12:15 p.m.

New subject: Install fonts on MS Windows

Henning Hraban Ramm schrieb am 23.09.2023 um 20:05:

...

Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:

...
Dear list, I have a very simple question, perhaps too simple... I've installed CTX on MSWindows x64, but I can't use the fonts I've installed, despite the command: set OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

That should be ok.

There is no need to set OSFONTDIR on Windows to get system fonts or fonts which are placed in the local TeX directory in the home folder. Wolfgang

Jean-Pierre Delange

12:37 p.m.

New subject: Install fonts on MS Windows

Thanks to Pablo, Hraban and Wolfgang for their advice. Precisely, I've never had any problems before with installing additional fonts for CTX with MS Windows : it usually "works". That's why I was wondering... So I'm going to put the extra fonts in /texmf-fonts/fonts/data as Hraban advises. But to make a "set OSFONTDIR=c:\windows\fonts" is allways necessary I presume (in order to make CTX able to use windows fonts) ? Many thanks ! JP Le 23/09/2023 à 20:15, Wolfgang Schuster a écrit :

...

Henning Hraban Ramm schrieb am 23.09.2023 um 20:05:

...
Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:

...
Dear list, I have a very simple question, perhaps too simple... I've installed CTX on MSWindows x64, but I can't use the fonts I've installed, despite the command: set OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

That should be ok.

There is no need to set OSFONTDIR on Windows to get system fonts or fonts which are placed in the local TeX directory in the home folder.

Wolfgang

___________________________________________________________________________________

If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context webpage : https://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : https://contextgarden.net ___________________________________________________________________________________

-- Jean-Pierre Delange Ancients&Moderns Professeur Agrégé de Philosophie (HC)

Wolfgang Schuster

12:56 p.m.

New subject: Install fonts on MS Windows

Jean-Pierre Delange schrieb am 23.09.2023 um 20:37:

...

Thanks to Pablo, Hraban and Wolfgang for their advice.

Precisely, I've never had any problems before with installing additional fonts for CTX with MS Windows : it usually "works".

That's why I was wondering... So I'm going to put the extra fonts in /texmf-fonts/fonts/data as Hraban advises.

But to make a "set OSFONTDIR=c:\windows\fonts" is allways necessary I presume (in order to make CTX able to use windows fonts) ?

The file mtxrun.lua contains the following check which sets default paths on Windows and Mac when no value for OSFONTDIR is set. do local osfontdir=osgetenv("OSFONTDIR") if osfontdir and osfontdir~="" then elseif osname=="windows" then ossetenv("OSFONTDIR","c:/windows/fonts//") elseif osname=="macosx" then ossetenv("OSFONTDIR","$HOME/Library/Fonts//;/Library/Fonts//;/System/Library/Fonts//") end end Wolfgang

Jean-Pierre Delange

12:30 p.m.

New subject: Install fonts on MS Windows

Dear Hraban, Even the general command to show all installed fonts : mtxrun --script fonts --list --all doesn't show the new fonts. I didn't use mtxrun --script font --reload --force I'll install fonts in /texmf-fonts/fonts/data/ directory as you say. Is it a good trick to create separate OTF and TTF etc. directories ? Merci beaucoup ! JP Le 23/09/2023 à 20:05, Henning Hraban Ramm a écrit :

...

Am 23.09.23 um 14:40 schrieb Jean-Pierre Delange:

...
Dear list, I have a very simple question, perhaps too simple... I've installed CTX on MSWindows x64, but I can't use the fonts I've installed, despite the command: set OSFONTDIR=c:\windows\fonts;c:\windows\users\adeimantos\context\texmf\fonts.

That should be ok.

...
I cleared the Windows font cache, restarted the machine, then reloaded with the command : mtxrun --script fonts --reload

That should be enough. Maybe try mtxrun --script font --reload --force I need that regularly for cambria.ttc to work (on OSX).

...
A search for the fonts I had installed (theano didot and gfs didot) gave no results with :

mtxrun --script fonts --list --pattern=*gfs* --all.

The pattern doesn’t need jokers. Did you try without?

...
I admit that I don't know in which directory it is best to install the fonts for optimum performance.

If you need the fonts also for other programs, the OS directory is good, otherwise <ConTeXt installation directory>/texmf-fonts/fonts/data/ is recommended.

Hraban ___________________________________________________________________________________

If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / https://www.ntg.nl/mailman/listinfo/ntg-context webpage : https://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : https://contextgarden.net ___________________________________________________________________________________

-- Jean-Pierre Delange Ancients&Moderns Professeur Agrégé de Philosophie (HC)

Henning Hraban Ramm

12:42 p.m.

New subject: Install fonts on MS Windows

Am 23.09.23 um 20:30 schrieb Jean-Pierre Delange:

...

Dear Hraban,

Even the general command to show all installed fonts :

mtxrun --script fonts --list --all

doesn't show the new fonts. I didn't use mtxrun --script font --reload --force

I'll install fonts in /texmf-fonts/fonts/data/ directory as you say.

Well, I don’t know if it will work if the OS font dir didn’t.

...

Is it a good trick to create separate OTF and TTF etc. directories ?

It’s not necessary. In 8bit times it made sense to separate metrics, outlines, mappings and stuff, but since OpenType fonts (.ttf or .otf doesn’t matter) contain everything, just order them as you like. According to TDS, a folder below fonts is expected, so Hans suggested "data", but you can use ttf, otf or wtf. Below that, you can use more folders to keep an overview. Hraban

Hamid,Idris

1:14 p.m.

------ Original Message ------ From "Hans Hagen via ntg-context" mailto:ntg-context@ntg.nl> To "ntg-context@ntg.nlmailto:ntg-context@ntg.nl" mailto:ntg-context@ntg.nl> Cc "Hans Hagen" mailto:j.hagen@freedom.nl> Date 9/22/2023 10:44:38 AM Subject [NTG-context] Re: Toggling the symbol for the zero-width joiner and related Unicode control characters (It seems he never considered making it an opentype feature in the font itself, but since his focus is/was XeTeX/HB (HB is rather rigid and dictatorial) I guess that's not surprising.) I admit that I don't follow what happens with xetex (they changed the rendere at some point indeed) not HB (I only notice that it gets updates frequently in the tex live repository which makes me wonder how one retains compatility unless one freezes). I actually kept the lib binding code that can use it around for your font testing (we wanted to see what uniscribe does), not sure if it still works. I remember the experimental uniscribe binding, don't recall that it was ever officially released. (Did Kai Bormann ever work with it?). I hope that it still works or can be resurrected when the time comes for testing -) Anyway, we're entering the bug cq. side effect becomes feature area here; just like yesterdays perfect bidi algorithm is todays less pefect one replaced by ... But therein lies the problem: ConTeXt shows the rendering by default, and we need to turn it off. Since most non-Latin typography targets Uniscribe applications which allows for toggling, the font developers (commercial or free) don't have to concern themselves with this issue. if context shows it then it is not a feature but hard coded shapes which is weird; how does one know what to 'remove' or not? And in what stage? If they are zero width it is simple to ignore them in the backend, if they have dimensions (w/h/d) then they contributed and wiping is tricky The metrics should be dimensionless and most Arabic fonts do that, but Lotus Linotype gives them width (one wonders why). Perhaps the Lotus designers thought that giving them width makes them easier to edit, which is indeed the case. And perhaps they were satisfied that Word/InDesign would be able, as needed, to toggle the metrics on and off as well as the shape. That WYSIWYG-DTP curse again.. \setcharacterstripping[1] gives the right results with Lotus (at least so far) -- see attached. Was pleasantly surprised that the spurious width was also suppressed. \setcharacterstripping[0] toggles back -- another pleasant result -) Since Word rules the world, most font designers target it. Since Word provides for toggling the symbols -- needed for editing purposes -- there was no need for Arabic-script font designers to worry about the symbols showing up where they are not wanted. (I suppose that InDesign behaves the same way.) I don't know ... irr these dtp programs are more like "if you want this feature applied select a range of characters and apply it" That's what was meant when I spoke of the continued effects of the WYSIWYG curse: It saved font designers from having to think much about this issue. In some way it's also flaws in the open type approach. Basically that happens when application stuff becomes a standard and one forgets that it was (is) application driven. (And you haven't seen variable fonts and color fonts yet ... no pretty standards either.) I'm in the process of switching from Multiple Masters sources to variable, so will surely encounter it soon. Unfortunately there is indeed a frustrating gray area in this arena between knowing where the standards end and where the diktats of corporations (or even free-software consortiums like HB) begin, as we've discussed in the past. With variable fonts, one has to add some degree of arbitrariness.. Not really -) This brings us to the point of consistency: For Arabic-script fonts, hard symbolic rendering of the Unicode control characters is the rule, not the exception. So not "an inconsistent mess" -- at least not as far as Arabic-script typography is concerned. Funny rules ... but I'm not going top enable 'wipe' by default: after all, one gets what one deserves, nto what one likes (which can differ per day). But you can enable the wiping. We can of course ignore in the backend when zero width but then how to explain that they contributed to the ht/dp (unless we wipe these dimensions) ... all slow-downers So far \setcharacterstripping[1] is giving the right results. If the relevant trackers are available, could add them to the attached test. But presumably you know how and why it works even for a non-dimensionless glyph. so you want to see soem zwj sumbol in a rendered text? Only in verbatim/\type'd text where it is appropriate, even necessary. Thanks to Word/WYSIWYG, the rule is de facto, but it is not de jure -) Ideally, Scintilla (Scite, Notepad++, etc.) should do the same, or provide a toggle, as MS Notepad does. (Tangent: In terms of Unicode functionality, MS Notepad is still unrivalled, even in 2023!) We agree that for final printed output it is not appropriate (except perhaps in a paper that discusses Unicode, fonts, etc., in which case it can be rendered using the figures or symbols mechanism -- or toggled as needed.) so what is it now: - for verbatim you can use almfixed and they show up (when they have a glyph) Except for \nobreakspace I think. At some point it may be worth seeing if there is a way to get the NBSP glyph to show in a typed buffer, or if the color-representation can be configured as some symbol instead of a color. But not urgent. - for other fonts if they have them they show up (unless gone in the process ot rendering) - but you can wipe them optionally not sure what more we need For now I think we're good. If anything comes up during the immediate project, will post it. Many thanks as always Hans. Best wishes -- Idris Samawi Hamid, Professor Department of Philosophy Colorado State University Fort Collins, CO 80523

297

Age (days ago)

299

Last active (days ago)

List overview

Download

20 comments

7 participants

participants (7)

Hamid,Idris
Hans Hagen
Hans Hagen
Henning Hraban Ramm
Jean-Pierre Delange
Pablo Rodriguez
Wolfgang Schuster

Toggling the symbol for the zero-width joiner and related Unicode control characters

tags

participants (7)