Hans Hagen wrote:

Marcin Borkowski wrote:

...

Hello!

I was just wondering, whether something like this is possible in luatex - and if not, if it might be implemented;). I'd like to be able to transform somehow the input right *before* TeX sees it. What I mean is e.g. a transformation using regexps. Application I have in mind is as follows: in Polish, there are a few prepositions which are one-letter words; it's a bad custom to leave them hanging at the end of a line. Such an option could automatically put ties (~) instead of spaces after them - a feature sometimes requested by Polish TeX users.

input parsing is not needed here, it's easier to to do this on the node list; remind me in a few months (it's rather trivial to implmenent since the infrastructure is already in mkiv)

Hans is right, the node list should be altered instead of the input, because you can never be sure that "\TEX\ i \METAFONT\" is not hidden inside a definition somewhere. There actually is a way to modify the input lines just before they are parsed by luatex, but that is really only suitable for converting 8-bit encodings to utf-8. (I was going to post an example abusing that method, but there are way many things that can go wrong, so I changed my mind) Best wishes, Taco

luigi scarso wrote:

...

There actually is a way to modify the input lines just before they are parsed by luatex, but that is really only suitable for converting 8-bit encodings to utf-8. (I was going to post an example abusing that method, but there are way many things that can go wrong, so I changed my mind) Post it anyway. We are all adults.

If you say so ... % engine=luatex \startluacode function add_ties (line) return line:gsub(' ','~') end \stopluacode \def\startfiltered {\ctxlua {callback.register('process_input_buffer', add_ties)}} \def\stopfiltered {\ctxlua {callback.register('process_input_buffer', nil)}} \starttext \startfiltered Thus, I came to the conclusion that the designer of a new system must not only be the implementer and first large||scale user; the designer should also write the first user manual. \stopfiltered \stoptext Notice how it breaks only at hyphens? If you think that is because those are the only valid breakpoints, you are actually slightly wrong: there are still 'normal' spaces between 'system' and 'must' and between 'the' and 'designer' (because EOL is converted to a space *after* reading in the line buffer). The lua function is _much_ too simple. It will happily convert '\ ' into '\~', $a = 2~$ into $a~=~2~$, and "\TEX is" into "\TEX~is". Best wishes, Taco

Olivier Guéry wrote:

On Fri, Apr 4, 2008 at 10:52 AM, Taco Hoekwater wrote:

...

luigi scarso wrote:

...

...

There actually is a way to modify the input lines just before they are parsed by luatex, but that is really only suitable for converting 8-bit encodings to utf-8. (I was going to post an example abusing that method, but there are way many things that can go wrong, so I changed my mind) Post it anyway. We are all adults.

If you say so ...

% engine=luatex

\startluacode function add_ties (line) return line:gsub(' ','~') end \stopluacode

Hello, new to this list, I see this first message. Is it a solution for my request (http://wiki.contextgarden.net/Feature_Requests) about french typography ?

\setcharacterspacing [frenchpunctuation] ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Fri, Apr 4, 2008 at 1:02 PM, Olivier Guéry wrote:

...

...

Hello, new to this list, I see this first message. Is it a solution for my request (http://wiki.contextgarden.net/Feature_Requests) about french typography ?

\setcharacterspacing [frenchpunctuation]

Thank's a lot. It's great !

I Test it with this :

Test; test: test? test! \crlf %english input Test : test ; test ? test ! \crlf %french input Test ; test : test ? test ! \crlf %french with nbsp

Et seems that only the english input get the spacing. Don't know why but the sign « ; » of the english input don't get the spacing. You put the value to .25em, looking in the french extension (who can't play with utf8) there's a .16667em value. It's seem's that the smallest is better because the caracter belong to the word, he's not between the two words.

You could also try to use the MkIV version. from core-spa.mkiv % experimental spacing % % test: oeps {\setcharacterspacing[frenchpunctuation]x: xx \bfd x: xx} oeps: test Wolfgang

On Fri, Apr 04 2008, Olivier Guéry wrote:

...

You could also try to use the MkIV version.

from core-spa.mkiv

% experimental spacing % % test: oeps {\setcharacterspacing[frenchpunctuation]x: xx \bfd x: xx} oeps: test

I see this when I search in the file but don't know how to try this… (I'm not a software ingineer at all, but a typography fan). It fact the « problem » for the moment is that I put space (and nbsp) in all my source documents (as all the french people, I think)… And it seems that ConTeXt can't deal with this for the moment.

It seems, that ConTeXt deals with that very well (attention, the following code is utf8): % engine=luatex \starttext \hsize1mm nbsp : test thinspace : test normalspace : test \stoptext Between "nbsp" and ":" is a nonbreakable space, and indeed there is no line-break. Between "thinspace" and ":" is a nonbreakable thin-space, and also no line-break. Between "normalspace" and ":" is a normal space, so there is a line-break. Does this look like what you want? Cheers, Peter -- http://pmrb.free.fr/contact/

On Fri, Apr 04 2008, Olivier Guéry wrote:

The problem is that even if the space before « : » is not a nbsp, it should be a thin nbsp (only silly guys like me change their layout to have nbsp on Shift+space ;o)).

Hello, Just my personal opinion: I always write these characters without any space before, because: - more space, more to type - the space is a matter of typography, so up to ConTeXt and not to the author - in all other languages there is no space to type (so I keep my habits) - if I use a normal space, the word-wrapping of my editor (emacs) can introduce line-breaks there - using nbsp (shift-space) means even more keypresses Cheers, Peter P.S.: As a French guy, what do you think about t-french.tex? -- http://pmrb.free.fr/contact/

Olivier Guéry wrote:

...

...

Hello, new to this list, I see this first message. Is it a solution for my request (http://wiki.contextgarden.net/Feature_Requests) about french typography ?

\setcharacterspacing [frenchpunctuation]

Thank's a lot. It's great !

I Test it with this :

Test; test: test? test! \crlf %english input Test : test ; test ? test ! \crlf %french input Test ; test : test ? test ! \crlf %french with nbsp

Et seems that only the english input get the spacing. Don't know why but the sign « ; » of the english input don't get the spacing. You put the value to .25em, looking in the french extension (who can't play with utf8) there's a .16667em value. It's seem's that the smallest is better because the caracter belong to the word, he's not between the two words.

you have to be more explicit; for instance, what do you mean with 'french input' it's an attribute, not coupled to a language, but grouping should work ok; npsp is not intercepted (no problem to do that) i can add all kind of thins but then need precise specs + test files Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Olivier Guéry wrote:

Looking to my tests it seems that if there's a space in the source file, the spacing is not apply — and I guess the nbsp neither. So, what would be great is we had something that say : no mater what is in front of this punctuations (space, nbsp or nothing) just make them nbsp with the right spacing !

it's no problem to add avariant that treats nospace, space, nbsp as equel, and removes in that case the space/nbsp but if we support such a thing, we need a better specification; there are probably more space related chars that needs treatment then ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

but if we support such a thing, we need a better specification; there are probably more space related chars that needs treatment then

There's not much to it, actually. There are some space characters in Unicode, and we should handle them as much Unicode-compliantly as possible; and there are some particular typographic conventions on top of that, for each language. It's easy to come up with a simple scheme to support both, and I've already outlined it on a different mailing-list (http://tug.org/pipermail/xetex/2008-February/008529.html). The problem is to decide how much of the users' old typing habits we want to take in account (and, Olivier, another message by me in the same thread addresses the exact same issues you were raising a few days ago: http://tug.org/pipermail/xetex/2008-February/008533.html). I can discuss that with you at BachoTeX, Hans. It's best done around a beer or two, anyway ;-) Arthur

On Wed, Apr 16, 2008 at 10:50 AM, Hans Hagen wrote:

Arthur Reutenauer wrote:

...

...

but if we support such a thing, we need a better specification; there are probably more space related chars that needs treatment then

There's not much to it, actually. There are some space characters in Unicode, and we should handle them as much Unicode-compliantly as possible; and there are some particular typographic conventions on top of that, for each language. It's easy to come up with a simple scheme to support both, and I've already outlined it on a different mailing-list (http://tug.org/pipermail/xetex/2008-February/008529.html). The problem is to decide how much of the users' old typing habits we

this is indeed an important point ... we don't want to cripple default behaviour by that (we already have -- --- and such)

The spaces before « ; » and others are not the same habits than « -- » and « --- ». « -- » is TeX specific, the spaces no, it's just a french habit.

the problem with all these automatisms is that it then becomes impossible to do something verbatim, i.e bypass those mechamisms

I can't imagine a french text without those spaces. So maybe this specific question should be on for all french text, and something like \setcharacterspacing [frenchpunctuation] [no] can be use for the excepts. (For sure questions like always indenting the paragraphs are more complex since modern typography sometime don't put them).

...

I can discuss that with you at BachoTeX, Hans. It's best done around a beer or two, anyway ;-)

sure, enough beer at bachotek anyway ...

If it can be solve with a beer ;o) Olivier. -- [Message tapé sur un clavier Bépo : http://www.clavier-dvorak.org ] Olivier nemolivier@gmail.com http://nemolivier.blogspot.com

Peter Münster wrote:

On Wed, Apr 16 2008, Olivier Guéry wrote:

...

I can't imagine a french text without those spaces. So maybe this specific question should be on for all french text, and something like \setcharacterspacing [frenchpunctuation] [no] can be use for the excepts.

There is already "\setcharacterspacing[reset]", but it does not work: ERROR: Missing number, treated as zero.

Until a bug-fix, you can use "\doresetattribute{spacing}".

I use for example \setuptyping[style=\doresetattribute{spacing}\tt] \setuptype[style=\doresetattribute{spacing}\tt] at some places.

i didn't know that this was broken .. fixed ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Fri, Apr 4, 2008 at 10:52 AM, Taco Hoekwater wrote:

luigi scarso wrote:

...

...

There actually is a way to modify the input lines just before they are parsed by luatex, but that is really only suitable for converting 8-bit encodings to utf-8. (I was going to post an example abusing that method, but there are way many things that can go wrong, so I changed my mind) Post it anyway. We are all adults.

If you say so ...

% engine=luatex

\startluacode function add_ties (line) return line:gsub(' ','~') end \stopluacode

\def\startfiltered {\ctxlua {callback.register('process_input_buffer', add_ties)}} \def\stopfiltered {\ctxlua {callback.register('process_input_buffer', nil)}}

\starttext \startfiltered Thus, I came to the conclusion that the designer of a new system must not only be the implementer and first large||scale user; the designer should also write the first user manual. \stopfiltered \stoptext

Notice how it breaks only at hyphens? If you think that is because those are the only valid breakpoints, you are actually slightly wrong: there are still 'normal' spaces between 'system' and 'must' and between 'the' and 'designer' (because EOL is converted to a space *after* reading in the line buffer).

The lua function is _much_ too simple. It will happily convert '\ ' into '\~', $a = 2~$ into $a~=~2~$, and "\TEX is" into "\TEX~is".

ok, many thanks. -- luigi it's new . it's powerful . it's luatex . http://www.luatex.org

On Fri, 4 Apr 2008, Peter Münster wrote:

On Fri, Apr 04 2008, Taco Hoekwater wrote:

...

% engine=luatex

\startluacode function add_ties (line) return line:gsub(' ','~') end \stopluacode

\def\startfiltered {\ctxlua {callback.register('process_input_buffer', add_ties)}} \def\stopfiltered {\ctxlua {callback.register('process_input_buffer', nil)}}

\starttext \startfiltered Thus, I came to the conclusion that the designer of a new system must not only be the implementer and first large||scale user; the designer should also write the first user manual. \stopfiltered \stoptext

Notice how it breaks only at hyphens? If you think that is because those are the only valid breakpoints, you are actually slightly wrong: there are still 'normal' spaces between 'system' and 'must' and between 'the' and 'designer' (because EOL is converted to a space *after* reading in the line buffer).

The lua function is _much_ too simple. It will happily convert '\ ' into '\~', $a = 2~$ into $a~=~2~$, and "\TEX is" into "\TEX~is".

Hello,

Perhaps this one could be already useable for the OP:

function add_ties(line) line = line:gsub('( %a) ', '%1~') line = line:gsub('^(%a) ', '%1~') line = line:gsub('( %a)$', '%1~%%') return line end

Not working: $a = b + c$.

How about substituting with a macro \nonbreakbablespace (instead of ~) and redefining \nonbreakablespace to \relax using \everymathematics or \mathortext? Aditya

Aditya Mahajan wrote:

...

Perhaps this one could be already useable for the OP:

function add_ties(line) line = line:gsub('( %a) ', '%1~') line = line:gsub('^(%a) ', '%1~') line = line:gsub('( %a)$', '%1~%%') return line end

Not working: $a = b + c$.

How about substituting with a macro \nonbreakbablespace (instead of ~) and redefining \nonbreakablespace to \relax using \everymathematics or \mathortext?

and then ending up with the same mess as usual ... exceptions, exceptions, exceptions ... just think of boxed text inside math which then needs to have it enabled again really, node lists is the way to go (although i admit that currently i do not handle math (also because it's a big todo in luatex) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

Dnia Fri, Apr 04, 2008 at 09:50:55AM +0200, Hans Hagen napisał(a):

Marcin Borkowski wrote:

...

Hello!

I was just wondering, whether something like this is possible in luatex - and if not, if it might be implemented;). I'd like to be able to transform somehow the input right *before* TeX sees it. What I mean is e.g. a transformation using regexps. Application I have in mind is as follows: in Polish, there are a few prepositions which are one-letter words; it's a bad custom to leave them hanging at the end of a line. Such an option could automatically put ties (~) instead of spaces after them - a feature sometimes requested by Polish TeX users.

input parsing is not needed here, it's easier to to do this on the node list; remind me in a few months (it's rather trivial to implmenent since the infrastructure is already in mkiv)

Thanks a lot;). I've set up a reminder in 3 months' time;).

Hans

-- Marcin Borkowski (http://mbork.faculty.fmcs.amu.edu.pl)