[reshufling the order a bit] First of all - I forgot to express my positive surprise - all the horizontal breaks including hyphenation are *exactly the same* in MKII and MKIV all over the document, even though MKII and MKIV are using different fonts. Which is really nice. \startyoumayignorethis The only thing that really bothers me are so much different vertical differences which show up in almost every document I ever write. MKIV consistently takes more vertical space, so no matter what document I try to recompile, it always ends up with more pages (and some weird page breaks) when processing it with MKIV. Yes, I know. The two are incompatible. Yes, I know. Different fonts are being used. Yes, I know. Different algorithms/aproaches to break pages. But it would be soooooo nice if the most basic documents with LM fonts could turn out at least approximately the same. Horizontal breaking works perfectly (it's identical). It's only vertical spacing that's a tiny bit "problematic" and makes every recompiled document a bit messy. It might be nice to look a bit closer to the topic, like in the example that I pointed out. There are some weirdnesses left, like the equations that starts at least a line lower in MKIV when there's no real reason for such a behaviour. \stopyoumayignorethis

...

\definedescription   [latexdesc] [headstyle={\ss\bf},style=normal,align=left,location=hanging,width=fit,margin=0cm]

just omit the align=left (was not handled in mkii)

Honestly, I have no idea why I had it there, but I guess that I was copy-pasting from wiki. So I didn't even notice that that option had no effect at all.

also, commands starting with \latex behave unpredictable in mkiv due to luigis compatibility mode

That was the best comment in the thread. Thanks for making me a day :)

...

\usetypescript[iwona] \setupbodyfont[iwona] \starttext $a+b$ \stoptext

hmm runs ok here ... but as i removed some old iwona stuff from type-otf

Thanks. In the new beta warnings are gone indeed. (I'll check the changes later.)

...

2.) Recently discussed on the mailing list - stopper has no influence in MKIV:

\setupcaptions[stopper={:}] \starttext \placefigure{title}{\framed{bla}} \stoptext

subtle difference:

\setupcaptions[numberstopper={:}] \starttext \placefigure{title}{\framed{bla}} \stoptext

all stopper, separator etc things are now more explicit as we have more control

Thanks a lot. (Just wandering - should stopper be a synonym for numberstopper in MKIV then or is stopper also used somewhere else?)

...

4.) A slightly more basic example:

\enableregime[utf-8] \starttext \startbuffer[gnu] \startMPcode draw \sometxt{čšž}; \stopMPcode \stopbuffer čšž \getbuffer[gnu] \stoptext

hm, i get a pdf file with

čšž čšž

so what happens at your end?

Did you try to run that with MKII? MKIV works fine (I know that the title was a bit misleading). I get the characters from font that are equal to the second byte of UTF-8 representation of input character.

...

5.) weird indeed, must be something basis as $$111111111111111111111111111111111111111111111111111111111111111111111111111111\normalreqno{!!}$$

also has it .. taco just confirmed that it must be something in luatex itself

Thanks. I'm sorry that I didn't simplify that one (I just took a random equation out of a document since it looked nicer).

...

6.) I know that \NR is not the most appropriate way to go into next row, but the following gives considerably different results in MKII and MKIV:

\starttable[|l|] \NC \type{a} \NC\NR \NC \type{a} \NC\NR \NC \type{a} \NC\NR \stoptable

probably because \type has no strut in mkiv ... maybe it should have

I don't know.

...

9.) Note the big difference of when the formula starts vertically on page:

\starttext \startformula a+b \stopformula \stoptext

interesting, i need to look into that

Thanks :)

...

11.) Missing bibliography entries:

\usemodule[bib] % this line spoils the show \setuppublications[alternative=num]

\setuppublicationlayout[webpage]{%        \inserttitle{\bgroup\it }{\egroup. }{}%        \inserturl{}{}{}% }

\startpublication        [k=FAIR,t=webpage,u=http://www.gsi.de/fair/]        \biburl{http://www.gsi.de/fair/}        \title{FAIR -- Facility for Antiproton and Ion Research} \stoppublication

\starttext \cite[FAIR]\par \placepublications \stoptext

magic to me, but we can look into bib mess stuff later

What can I do?

...

3.) Bibliography citations give different result with the same setup (different numbers and different spacing in \placepublication - might be also worth testing with grid typesetting as I wasn't able to set up the alignment properly in MKII last time when I have tried). I'm not sure which one is right, but there's probably no reason to produce different results.

\usemodule        [bib] \setuppublications        [alternative=num,         criterium=all,         sorttype=cite]

\starttext \startpublication[k=a,t=article]\arttitle{a}\stoppublication \startpublication[k=b,t=article]\arttitle{b}\stoppublication \startpublication[k=c,t=article]\arttitle{c}\stoppublication

x\cite[a,b,c]

\placepublications \stoptext

as i never used citations i don't know what you expect ... best ask Thomas what you should expect ... anyway, i'll make a template for the xml approach

Does anyone else have an idea of whether \cite[a,b,c] should generate [1,2,3] or [1-3]? I have an article at hand that has "[1], [2], [3]" instead to be honest. I'm not sure, but [1-3] somehow doesn't appear right to me. I have a feeling that each bib item needs to be cited separately, but I may be wrong. Mojca

On 13-5-2010 3:42, Mojca Miklavec wrote:

\startyoumayignorethis The only thing that really bothers me are so much different vertical differences which show up in almost every document I ever write. MKIV consistently takes more vertical space, so no matter what document I try to recompile, it always ends up with more pages (and some weird page breaks) when processing it with MKIV.

\startcantresistmode the lineheight relates to the ex height and as in mkiv we don't have the tfm limitations (those 16 values of ht dp) we have slightly different spacing \stopcantresistmode

Yes, I know. The two are incompatible. Yes, I know. Different fonts are being used. Yes, I know. Different algorithms/aproaches to break pages. But it would be soooooo nice if the most basic documents with LM fonts could turn out at least approximately the same. Horizontal breaking works perfectly (it's identical). It's only vertical spacing that's a tiny bit "problematic" and makes every recompiled document a bit messy.

\startchallengingmode well, you can try to figure out what 2.8ex in mkii and mkiv is and then have your own defaults for mkiv \stopchallengingmode

It might be nice to look a bit closer to the topic, like in the example that I pointed out. There are some weirdnesses left, like the equations that starts at least a line lower in MKIV when there's no real reason for such a behaviour.

\startexplanation TeX tries hard to inject a baselineskip and also an empty hlist so that one always gets that line. In MkII I compensated for that hard to beat automatism. In MkIV this does not happen. We figured out that when we add a \noindent before $$ that we don't get this side effect so that now happens in the latest beta. \stopexplanation

\stopyoumayignorethis

Thanks a lot. (Just wandering - should stopper be a synonym for numberstopper in MKIV then or is stopper also used somewhere else?)

the more synonyme the more documentation

Did you try to run that with MKII? MKIV works fine (I know that the title was a bit misleading). I get the characters from font that are equal to the second byte of UTF-8 representation of input character.

you probably need to enable utf8 in the mp environment then

Thanks. I'm sorry that I didn't simplify that one (I just took a random equation out of a document since it looked nicer).

random thesis ... interesting

...

...

9.) Note the big difference of when the formula starts vertically on page:

\starttext \startformula a+b \stopformula \stoptext

interesting, i need to look into that

Thanks :)

same problem as previous

What can I do?

make a complete test as small as possible

Does anyone else have an idea of whether \cite[a,b,c] should generate [1,2,3] or [1-3]? I have an article at hand that has "[1], [2], [3]" instead to be honest. I'm not sure, but [1-3] somehow doesn't appear right to me. I have a feeling that each bib item needs to be cited separately, but I may be wrong.

collapsing has always been there afaik Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 13-5-2010 4:32, Khaled Hosny wrote:

On Thu, May 13, 2010 at 04:16:12PM +0200, Hans Hagen wrote:

...

On 13-5-2010 3:42, Mojca Miklavec wrote:

...

\startyoumayignorethis The only thing that really bothers me are so much different vertical differences which show up in almost every document I ever write. MKIV consistently takes more vertical space, so no matter what document I try to recompile, it always ends up with more pages (and some weird page breaks) when processing it with MKIV.

\startcantresistmode

the lineheight relates to the ex height and as in mkiv we don't have the tfm limitations (those 16 values of ht dp) we have slightly different spacing

Something I find very annoying is variable interline spacing, if I've, for example, a line with some Arabic words vocalized I get some times too much white space above it that it almost looks like an empty line. It makes the page look like crap. Is there a way to force fixed interline spacing?

turn turn grid on .. but even then, we need some nice heuristic for determing the right ht/dp ratio for arabic (can be set up) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 13-5-2010 7:08, Khaled Hosny wrote:

On Thu, May 13, 2010 at 05:00:01PM +0200, Hans Hagen wrote:

...

On 13-5-2010 4:32, Khaled Hosny wrote:

...

On Thu, May 13, 2010 at 04:16:12PM +0200, Hans Hagen wrote:

...

On 13-5-2010 3:42, Mojca Miklavec wrote:

...

\startyoumayignorethis The only thing that really bothers me are so much different vertical differences which show up in almost every document I ever write. MKIV consistently takes more vertical space, so no matter what document I try to recompile, it always ends up with more pages (and some weird page breaks) when processing it with MKIV.

\startcantresistmode

the lineheight relates to the ex height and as in mkiv we don't have the tfm limitations (those 16 values of ht dp) we have slightly different spacing

Something I find very annoying is variable interline spacing, if I've, for example, a line with some Arabic words vocalized I get some times too much white space above it that it almost looks like an empty line. It makes the page look like crap. Is there a way to force fixed interline spacing?

turn turn grid on .. but even then, we need some nice heuristic for determing the right ht/dp ratio for arabic (can be set up)

I recall trying grid a while ago but it didn't work, looks like I have to set \setuplayout[grid=force].

for sure there are bugs as it needs much testing but we have predefined grid setups: % none don't enlarge % halfline enlarge by halfline/halfline % line enlarge by line/line % strut enlarge by ht/dp (default) % first align to top line % last align to bottom line % mindepth round depth down % maxdepth round depth up % minheight round height down % maxheight round height up % local use local interline space % shift:-3tp vertical shift within box \definegridsnapping[normal] [maxheight,maxdepth,strut] \definegridsnapping[standard] [maxheight,maxdepth,strut] \definegridsnapping[yes] [maxheight,maxdepth,strut] \definegridsnapping[strict] [\v!maxdepth:0.8,maxheight:0.8,strut] \definegridsnapping[tolerant] [\v!maxdepth:1.2,maxheight:1.2,strut] \definegridsnapping[top] [minheight,maxdepth,strut] \definegridsnapping[bottom] [maxheight,mindepth,strut] \definegridsnapping[both] [minheight,mindepth,strut] \definegridsnapping[broad] [maxheight,maxdepth,strut,0.8] \definegridsnapping[fit] [maxheight,maxdepth,strut,1.2] \definegridsnapping[first] [first] \definegridsnapping[last] [last] \definegridsnapping[high] [minheight,maxdepth,none] \definegridsnapping[low] [maxheight,mindepth,none] \definegridsnapping[line] [line] \definegridsnapping[strut] [strut] \definegridsnapping[max] [maxdepth,maxheight,strut] \definegridsnapping[min] [mindepth,minheight,strut] eventually an structural elements will have a grid key Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Thu, 13 May 2010 10:00:01 -0500, Hans Hagen wrote:

On 13-5-2010 4:32, Khaled Hosny wrote:

...

On Thu, May 13, 2010 at 04:16:12PM +0200, Hans Hagen wrote:

...

On 13-5-2010 3:42, Mojca Miklavec wrote:

...

\startyoumayignorethis The only thing that really bothers me are so much different vertical differences which show up in almost every document I ever write. MKIV consistently takes more vertical space, so no matter what document I try to recompile, it always ends up with more pages (and some weird page breaks) when processing it with MKIV.

\startcantresistmode

the lineheight relates to the ex height and as in mkiv we don't have the tfm limitations (those 16 values of ht dp) we have slightly different spacing

Something I find very annoying is variable interline spacing, if I've, for example, a line with some Arabic words vocalized I get some times too much white space above it that it almost looks like an empty line. It makes the page look like crap. Is there a way to force fixed interline spacing?

Can you give an example?

turn turn grid on .. but even then, we need some nice heuristic for determing the right ht/dp ratio for arabic (can be set up)

I have some nice texts that illustrate a standard balance, but I'd like to see what Khaled has in mind exactly before I comment further... Peace Idris -- Professor Idris Samawi Hamid, Editor-in-Chief International Journal of Shi`i Studies Department of Philosophy Colorado State University Fort Collins, CO 80523

On Thu, 13 May 2010 15:06:09 -0500, Khaled Hosny wrote:

...

...

...

Something I find very annoying is variable interline spacing, if I've, for example, a line with some Arabic words vocalized I get some times too much white space above it that it almost looks like an empty line. It makes the page look like crap. Is there a way to force fixed interline spacing?

Can you give an example?

...

turn turn grid on .. but even then, we need some nice heuristic for determing the right ht/dp ratio for arabic (can be set up)

I have some nice texts that illustrate a standard balance, but I'd like to see what Khaled has in mind exactly before I comment further... Nothing special, I always expect interline space to be fixed, I don't know if TeX always make interline spacing variable, but this wasn't an issue with English text. However, with Arabic, Tashkil marks seems to always cause a noticeable extra whitespace above the line. See the uneven distribution of vertical whitespace in this example (it can be even worse than this in reality): \usemodule[simplefonts] \setmainfont[Arabic Typesetting][features=arabic] \starttext \pardir TRT\textdir TRT \dorecurse{10}{\dorecurse{20}{نص عربي } نَصُّ مُشكَّل \dorecurse{20}{نص عربي}} \stoptext

I used to find this annoying years ago but today I look at it as a good feature. In your \definebodyfontenvironment you just have to define a good interlinespace. Presumably the one picked up from the font by luatex is not high enough. In my journal, I used to use \struttedbox's (or something similarly named) to suppress this when mixing english and arabic... Professionally mixed latin-arabic texts often adjust the interline spacing, especially when arabic is introduced into latin paragraphs. This is almost unavoidable since making arabic-script readable and matchable to latin involves a larger relative size for aesthetic optics. Sometimes forcing will look nice, but even then one probably has to add a bit of interlinespace to the latin font to get the right balance. That is, a latin document that uses a LOT of interparagraph arabic will want a bit extra interlinespace to begin with, so that "forcing" will generally give good results. Best wishes Idris -- Professor Idris Samawi Hamid, Editor-in-Chief International Journal of Shi`i Studies Department of Philosophy Colorado State University Fort Collins, CO 80523

On 13-5-2010 10:06, Khaled Hosny wrote:

Nothing special, I always expect interline space to be fixed, I don't know if TeX always make interline spacing variable, but this wasn't an issue with English text. However, with Arabic, Tashkil marks seems to always cause a noticeable extra whitespace above the line. See the uneven distribution of vertical whitespace in this example (it can be even worse than this in reality):

for arabic you really need to set the interline space (idris might have more input on this) - it has more height than depth - vowels add to the height of the line - tex adds some interline space when lines touch

\usemodule[simplefonts] \setmainfont[Arabic Typesetting][features=arabic]

\starttext \pardir TRT\textdir TRT \dorecurse{10}{\dorecurse{20}{نص عربي } نَصُّ مُشكَّل \dorecurse{20}{نص عربي}} \stoptext

Regards, Khaled

-- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 13-5-2010 11:26, Khaled Hosny wrote:

On Thu, May 13, 2010 at 10:25:06PM +0200, Hans Hagen wrote:

...

On 13-5-2010 10:06, Khaled Hosny wrote:

...

Nothing special, I always expect interline space to be fixed, I don't know if TeX always make interline spacing variable, but this wasn't an issue with English text. However, with Arabic, Tashkil marks seems to always cause a noticeable extra whitespace above the line. See the uneven distribution of vertical whitespace in this example (it can be even worse than this in reality):

for arabic you really need to set the interline space (idris might have more input on this)

- it has more height than depth

Not always عٍ or فيٍ is as deep as high is أً.

so for say 12pt arabic we should use 8pt ht and 8pt depth? ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 14-5-2010 2:37, Khaled Hosny wrote:

On Thu, May 13, 2010 at 11:35:58PM +0200, Hans Hagen wrote:

...

On 13-5-2010 11:26, Khaled Hosny wrote:

...

On Thu, May 13, 2010 at 10:25:06PM +0200, Hans Hagen wrote:

...

On 13-5-2010 10:06, Khaled Hosny wrote:

...

Nothing special, I always expect interline space to be fixed, I don't know if TeX always make interline spacing variable, but this wasn't an issue with English text. However, with Arabic, Tashkil marks seems to always cause a noticeable extra whitespace above the line. See the uneven distribution of vertical whitespace in this example (it can be even worse than this in reality):

for arabic you really need to set the interline space (idris might have more input on this)

- it has more height than depth

Not always عٍ or فيٍ is as deep as high is أً.

so for say 12pt arabic we should use 8pt ht and 8pt depth?

I'm not sure about this, but I'd rather trust the font designer for knowing better about his font, and something like in the attached file (except it doesn't work :) see my other mail)

it's a gray area ... if we obey the ht/dp spec of a font (ascender and descender) we end up with an unpredictable mess (actually comparable to setting the baselineskip to 0pt and trusting the ht/dp of glyphs to force the right line spacing) in dtp there's aways this rather strict grid Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Fri, 14 May 2010 06:37:02 -0600, Khaled Hosny wrote:

...

...

...

for arabic you really need to set the interline space (idris might have more input on this)

- it has more height than depth

Not always عٍ or فيٍ is as deep as high is أً.

so for say 12pt arabic we should use 8pt ht and 8pt depth?

I'm not sure about this, but I'd rather trust the font designer for knowing better about his font, and something like in the attached file (except it doesn't work :) see my other mail)

systems : begin file linespace.tex at line 7 ! You can't use `\dimexpr' in horizontal mode. <recently read> \dimexpr \setstrutgridyes ...e \strutheightfactor \dimexpr \normallineheight \fi \str... \setstrut ->\ifgridsnapping \setstrutgridyes \else \setstrutgridnop \fi <inserted text> ...nizefont \fi \spacing \plusone \presetnormallineheight \s... \setfontparameters ...tsfalse \the \everybodyfont \synchronizefontstrue \dosetupspecifiedinterlinespaceindeed ...rameters \updateraggedskips ... l.24 \stopluacode ? In any case, overly trusting the font designer is a bad idea in this arena, particularly with Arabic fonts. For example, vowels are treated as something of an afterthought even in "good" arabic fonts; yet they affect linespace calculations. In a document with no vowels one can get away with things you cannot get away with when they are extensively used; manual control is essential. And "good" arabic fonts contain numerous errors anyway. This is part of the luatex/mkiv philosophy anyway; otherwise we'd just use uniscribe or pango ;-) Peace Idris -- Professor Idris Samawi Hamid, Editor-in-Chief International Journal of Shi`i Studies Department of Philosophy Colorado State University Fort Collins, CO 80523

On 13-5-2010 8:34, Mojca Miklavec wrote:

Dear Hans,

(I'll answer the rest later)

...

...

Did you try to run that with MKII? MKIV works fine (I know that the title was a bit misleading). I get the characters from font that are equal to the second byte of UTF-8 representation of input character.

you probably need to enable utf8 in the mp environment then

But it has always worked perfectly until recently. And it's not really mp environment. I suspect that the text gets typeset with TeX, not with metapost, so metapost enviroment should not really count.

I have a version of ConTeXt that dates back to 2009.11.24 10:13 and in that one it still works properly. In version 2010.02.26 10:57 it is broken.

hm, let's move this off list then ... (just make very small for sure failing tests that i then can add to the test suite)

You know ... you have written a devoted \startsimplethesis for me, don't you remember :) Random images and equations were included there as well, I just extended macros a bit to adapt the result to the expertise of each jury member last september :)

surely i remember and i'm working on a more advanced version for your upcoming thesis, with automatic bibliographic references and so, but i'll only finish it when you will use mkiv

...

...

What can I do?

make a complete test as small as possible

It doesn't get much simpler than the following:

\usemodule[bib] % this line spoils the show

% this example spoils my weekend

\setuppublications[alternative=num] \setuppublicationlayout[webpage]{\inserturl{}{}{}} \startpublication[k=pragma,t=webpage] \biburl{http://www.pragma-ade.com} \stoppublication

\starttext \cite[pragma]\par\placepublications \stoptext

hm, you set it to num and get a num only? i'll have to check what happens Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On Wed, 12 May 2010, Taco Hoekwater wrote:

Aditya Mahajan wrote:

...

...

10.) Different square root shape/variant used:

$\frac{1}{\sqrt{2πσ_i^2}}$

Original formula: \startformula \startcases \NC ρ_0, \NC for $i=0$, \NR \NC ρ_i\,\frac{1}{\sqrt{2πσ_i^2}} \exp\biggl(-\frac{\left(x_j-μ_i\right)^2}{2σ_i^2}\biggr), \NC for $i\in\left\{1,2\right\}$. \NR \stopcases \stopformula but I cannot reproduce the big difference it makes in original document on a smaller scale.

Taco, is there a difference between the default math parameters for MkIV?

Hans now uses OpenType Math (sometimes faked) fonts always, and his initial version of the reimplemenation of \big.... was wrong. But I thought this had been fixed a few weeks ago.

I am using a month old version. I'll update and check again. Mojca's question was regarding the shape of \sqrt. Aditya