On 15 January 2016 at 17:34, Thomas A. Schmitz wrote:

On 01/15/2016 06:21 PM, Kate F wrote:

...

Hi. I'm using MkIV's XML stuff. I have two related questions.

Firstly, can I have entities in XML expand out to TeX markup?

I thought this was how you'd do it, but this produces a document that says literally "xxx~yyy\thinspace zzz", whereas I want the ~ and \thinspace to be executed as TeX commands. Test case below.

Secondly, is it possible to have ConTeXt load such entities from a DTD? If I use a DOCTYPE in my XML, entities from that DTD seem to not be loaded.

What do you mean? The way ConText processes entities is completely independent from your DTD.

So I see! But many DTDs contain definitions for entities, which I would like to use, rather than repeating those definitions by \xmlsetentity in ConTeXt. Some XML documents also have their own document-specific entities inside the DOCTYPE at the top of the file. For example in legal agreements, I have something like: <?xml version="1.0"?> <!DOCTYPE blah SYSTEM "blah.dtd" [ <!ENTITY us "the company"> <!ENTITY you "the customer"> ]> Where the stuff inside [...] is considered part of a document-specific DTD. libxml2 handles this sort of thing with its "dtdattr" options; I presume lxml has something similar.

...

a.xml: <?xml version="1.0"?> <a>xxx yyy zzz</a>

a.tex: \startxmlsetups xml:mysetups \xmlsetsetup{\xmldocument}{a}{xml:*} \stopxmlsetups \xmlregistersetup{xml:mysetups}

\startxmlsetups xml:a \xmlflush{#1} \stopxmlsetups

\enabledirectives[lxml.escapedentities] \xmltexentity{nbsp}{~} \xmltexentity{thinsp}{\thinspace}

So that's how to use \xmltexentity! I had seen that, but couldn't find how to use it. Thanks! -- Kate

On 1/15/2016 9:20 PM, Thomas A. Schmitz wrote:

On 01/15/2016 06:58 PM, Kate F wrote:

...

So I see! But many DTDs contain definitions for entities, which I would like to use, rather than repeating those definitions by \xmlsetentity in ConTeXt. Some XML documents also have their own document-specific entities inside the DOCTYPE at the top of the file.

For example in legal agreements, I have something like:

<?xml version="1.0"?> <!DOCTYPE blah SYSTEM "blah.dtd" [ <!ENTITY us "the company"> <!ENTITY you "the customer"> ]>

Where the stuff inside [...] is considered part of a document-specific DTD. libxml2 handles this sort of thing with its "dtdattr" options; I presume lxml has something similar.

AFAIK, ConTeXt has its own xml parser based on lua lpeg, see file lxml-tab.lua. There is code there that treats entities, but I have never used this approach, so you'll probably have to wait till Hans looks at your question (or understand the lua code...)

i'll come back to it ... fyi: these doctype entities are already parsed in mkiv .. just not used (or maybe it got lost when fulfilling other wishes) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 1/16/2016 4:55 PM, Kate F wrote:

On 16 January 2016 at 00:33, Hans Hagen wrote:

...

On 1/15/2016 9:20 PM, Thomas A. Schmitz wrote:

...

On 01/15/2016 06:58 PM, Kate F wrote:

...

So I see! But many DTDs contain definitions for entities, which I would like to use, rather than repeating those definitions by \xmlsetentity in ConTeXt. Some XML documents also have their own document-specific entities inside the DOCTYPE at the top of the file.

For example in legal agreements, I have something like:

<?xml version="1.0"?> <!DOCTYPE blah SYSTEM "blah.dtd" [ <!ENTITY us "the company"> <!ENTITY you "the customer"> ]>

Where the stuff inside [...] is considered part of a document-specific DTD. libxml2 handles this sort of thing with its "dtdattr" options; I presume lxml has something similar.

AFAIK, ConTeXt has its own xml parser based on lua lpeg, see file lxml-tab.lua. There is code there that treats entities, but I have never used this approach, so you'll probably have to wait till Hans looks at your question (or understand the lua code...)

i'll come back to it ... fyi: these doctype entities are already parsed in mkiv .. just not used (or maybe it got lost when fulfilling other wishes)

Much appreciated, if you would. Thank you! Is it just a case of having that parsing call \xmlsetentity?

it should work in the in beta now ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 18 January 2016 at 13:30, Thomas A. Schmitz wrote:

On 01/17/2016 07:24 PM, Hans Hagen wrote:

...

it should work in the in beta now

Wonderful! Thank you so much.

Hi Hans,

now I have a problem :-) What should take precedence if an entity is both defined in the dtd and as a \xmltexentity? The way I see it, the latter: e.g., in the DTD, I might declare something for use in a browser but require a different solution when typesetting with ConTeXt. The latest and greatest now takes my DTD definitions instead of the \xmltexentities, which did not happen before. Is that an unwanted side effect or the new default?

I was just about to ask the same! I agree that \xmltexentity ought to have precedence. I have an example: <!ENTITY nbsp "&#160;"> Which for some reason is being rendered as U+FFFD REPLACEMENT CHARACTER (a question-mark in a diamond). I don't know why that comes out as U+FFFD actually, since if I type a literal non-breaking space character instead, the PDF comes out just as if I'd used ~ in TeX. Maybe that's a bug somewhere? In any case, I think that sometimes it is helpful to override entities specifically for TeX processing, which aren't relevant for processing the XML source in other ways. -- Kate

On 1/18/2016 5:22 PM, Kate F wrote:

On 18 January 2016 at 13:30, Thomas A. Schmitz wrote:

...

On 01/17/2016 07:24 PM, Hans Hagen wrote:

...

it should work in the in beta now

Hi Hans,

now I have a problem :-) What should take precedence if an entity is both defined in the dtd and as a \xmltexentity? The way I see it, the latter: e.g., in the DTD, I might declare something for use in a browser but require a different solution when typesetting with ConTeXt. The latest and greatest now takes my DTD definitions instead of the \xmltexentities, which did not happen before. Is that an unwanted side effect or the new default?

Ah, there's a bug:

<!ENTITY i.opt "<option>-i</option>">

This should produce an <option> node in the DOM tree, just as if you'd typed that out where the entity is used. Currently ConTeXt takes that as literal text, as if you'd typed "<option>-i<option/>"

Often I wish XML weren't so complex...

are you sure? i've never seen that Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 1/18/2016 9:49 PM, Kate F wrote:

On 18 January 2016 at 19:13, Hans Hagen wrote:

...

On 1/18/2016 5:22 PM, Kate F wrote:

...

On 18 January 2016 at 13:30, Thomas A. Schmitz wrote:

...

On 01/17/2016 07:24 PM, Hans Hagen wrote:

...

it should work in the in beta now

Hi Hans,

now I have a problem :-) What should take precedence if an entity is both defined in the dtd and as a \xmltexentity? The way I see it, the latter: e.g., in the DTD, I might declare something for use in a browser but require a different solution when typesetting with ConTeXt. The latest and greatest now takes my DTD definitions instead of the \xmltexentities, which did not happen before. Is that an unwanted side effect or the new default?

Ah, there's a bug:

<!ENTITY i.opt "<option>-i</option>">

This should produce an <option> node in the DOM tree, just as if you'd typed that out where the entity is used. Currently ConTeXt takes that as literal text, as if you'd typed "<option>-i<option/>"

Often I wish XML weren't so complex...

are you sure? i've never seen that

Hans

Yep!

These are called "internal parsed entities". "Parsed" requires that any tags *inside* the entity must be balanced, unlike in SGML entities.

Sorry I can't find a clear explanation in the XML spec; it's a pretty confusing document. But here's some random person's slide illustrating an example: http://images.slideplayer.com/23/6622270/slides/slide_47.jpg

libxml2 deals with these correctly, which is what I've been using (xsltproc and friends) for my documents which use them. I generally trust libxml2 to get things right.

I use these entities to centralise often-repeated fragments between documents, kind of like how you might use a primitive macro in TeX.

So for example in one external DTD I have some general things:

<!ENTITY macro.arg "<replaceable>macro</replaceable>"> <!ENTITY equal.lit "<literal>=</literal>">

And then in one specific document's internal entities, something which uses them:

<!ENTITY D2.opt "<option>-D</option>¯o.arg;&equal.lit;<replaceable>defn</replaceable>">

Then if I change my mind about how I want to mark up "=", for example, I only have one place to change it. This makes life with XML a little bit less painful.

well, i've learned not to trust all these docs on the web too much and applications can do what they want (and thereby even influence standards) xml pocket reference: - parsed entity: replacement text that can be referenced internal: literal string to be injected (then the example shows only text and entities in your example you use a (decimal) character entity ... the link you give says that you cannot use & and % as part of the entities value so that would mean your example is wrong of course we can consider an option to parse the entity as xml (we can consider < as a trigger for parsing thereby kin dof automatically adapting) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 19 January 2016 at 02:16, Kate F wrote:

On 18 January 2016 at 21:16, Hans Hagen wrote:

...

On 1/18/2016 9:49 PM, Kate F wrote:

...

On 18 January 2016 at 19:13, Hans Hagen wrote:

...

On 1/18/2016 5:22 PM, Kate F wrote:

...

On 18 January 2016 at 13:30, Thomas A. Schmitz wrote:

...

On 01/17/2016 07:24 PM, Hans Hagen wrote: > > > > it should work in the in beta now

Hi Hans,

now I have a problem :-) What should take precedence if an entity is both defined in the dtd and as a \xmltexentity? The way I see it, the latter: e.g., in the DTD, I might declare something for use in a browser but require a different solution when typesetting with ConTeXt. The latest and greatest now takes my DTD definitions instead of the \xmltexentities, which did not happen before. Is that an unwanted side effect or the new default?

Ah, there's a bug:

<!ENTITY i.opt "<option>-i</option>">

This should produce an <option> node in the DOM tree, just as if you'd typed that out where the entity is used. Currently ConTeXt takes that as literal text, as if you'd typed "<option>-i<option/>"

Often I wish XML weren't so complex...

are you sure? i've never seen that

Hans

Yep!

These are called "internal parsed entities". "Parsed" requires that any tags *inside* the entity must be balanced, unlike in SGML entities.

Sorry I can't find a clear explanation in the XML spec; it's a pretty confusing document. But here's some random person's slide illustrating an example: http://images.slideplayer.com/23/6622270/slides/slide_47.jpg

libxml2 deals with these correctly, which is what I've been using (xsltproc and friends) for my documents which use them. I generally trust libxml2 to get things right.

I use these entities to centralise often-repeated fragments between documents, kind of like how you might use a primitive macro in TeX.

So for example in one external DTD I have some general things:

<!ENTITY macro.arg "<replaceable>macro</replaceable>"> <!ENTITY equal.lit "<literal>=</literal>">

And then in one specific document's internal entities, something which uses them:

"<option>-D</option>¯o.arg;&equal.lit;<replaceable>defn</replaceable>">

Then if I change my mind about how I want to mark up "=", for example, I only have one place to change it. This makes life with XML a little bit less painful.

well, i've learned not to trust all these docs on the web too much and applications can do what they want (and thereby even influence standards)

xml pocket reference:

- parsed entity: replacement text that can be referenced

internal: literal string to be injected (then the example shows only text and entities

in your example you use a (decimal) character entity ... the link you give says that you cannot use & and % as part of the entities value so that would mean your example is wrong

Indeed I did not mean that somebody's presentation slides are normative.

This is normative for XML 1.0: https://www.w3.org/TR/1998/REC-xml-19980210#wf-entities

Which has the following productions:

extParsedEnt ::= TextDecl? content content ::= (element | CharData | Reference | CDSect | PI | Comment)*

So you can see that both elements (by "element") and entity references (by "Reference") are permitted in the grammar. The latter includes:

Reference ::= EntityRef | CharRef EntityRef::='&' Name ';'

Sorry, I pointed to an outdated version of the XML 1.0 spec there. The current version (fifth revision) has the grammar written slightly differently, but still permits both well-formed <xyz>...</xyz> tags and &xyz; entities inside entity declarations: https://www.w3.org/TR/xml/#sec-entity-decl [70] EntityDecl ::= GEDecl | PEDecl [72] PEDecl ::= '<!ENTITY' S '%' S Name S PEDef S? '>' [74] PEDef ::= EntityValue | ExternalID [9] EntityValue ::= '"' ([^%&"] | PEReference | Reference)* '"' | "'" ([^%&'] | PEReference | Reference)* "'" [67] Reference ::= EntityRef | CharRef [68] EntityRef ::= '&' Name ';' Thus &xyz; is permitted by productions applying ultimately to EntityRef, and <xyz>...</xyz> is permitted by productions applying through the * closure of EntityValue's [^%&"] text (where [^...] means "not").

Per this, all my examples are correct.

O'Reilly XML in a Nutshell has an example: http://docstore.mik.ua/orelly/xml/xmlnut/ch03_04.htm

...

of course we can consider an option to parse the entity as xml

(we can consider < as a trigger for parsing thereby kin dof automatically adapting)

I do not think this is a good idea - per the XML spec, these entities should always be taken as well-formed fragments of XML. So treating them otherwise would be incorrect.

Thanks,

-- Kate

On 18 January 2016 at 20:07, Hans Hagen wrote:

On 1/18/2016 2:30 PM, Thomas A. Schmitz wrote:

...

On 01/17/2016 07:24 PM, Hans Hagen wrote:

...

it should work in the in beta now

Hi Hans,

now I have a problem :-) What should take precedence if an entity is

so you have an example ?

...

both defined in the dtd and as a \xmltexentity? The way I see it, the latter: e.g., in the DTD, I might declare something for use in a browser but require a different solution when typesetting with ConTeXt. The latest and greatest now takes my DTD definitions instead of the \xmltexentities, which did not happen before. Is that an unwanted side effect or the new default?

we probably have to make it configureable

I don't think it's necessary to have the precedence configurable - I think it would make sense to have \xmltexentity always take precedence. The usual case would be that there are no user-specified \xmltexentity definitions present at all, and the document's DTD entities would be used (by the feature you just added). So if somebody does have their own \xmltexentity specified, it's because they want that to be used instead. That would allow a user to do something specific for ConTeXt processing, but I don't have a good example for that. Personally I hope I can delete all my \xmltexentity calls now. -- Kate

On 1/18/2016 2:30 PM, Thomas A. Schmitz wrote:

On 01/17/2016 07:24 PM, Hans Hagen wrote:

...

it should work in the in beta now

Hi Hans,

now I have a problem :-) What should take precedence if an entity is both defined in the dtd and as a \xmltexentity? The way I see it, the latter: e.g., in the DTD, I might declare something for use in a browser but require a different solution when typesetting with ConTeXt. The latest and greatest now takes my DTD definitions instead of the \xmltexentities, which did not happen before. Is that an unwanted side effect or the new default?

i already feared that ... % \disabledirectives[lxml.entities.escaped] % \disabledirectives[lxml.entities.prefertex] \setupbodyfont[dejavu] \xmltexentity{oeps}{\textdegree} \startbuffer[test] <?xml version="1.0"?> <!DOCTYPE blah SYSTEM "blah.dtd" [ <!ENTITY oeps "indeed"> <!ENTITY more "&#160;"> ]> <text> Random text with an entity: &oeps; test & test Random text with an entity: &more; test </text> \stopbuffer \startxmlsetups xml:testsetups \xmlsetsetup{#1}{text}{xml:*} \stopxmlsetups \xmlregistersetup{xml:testsetups} \startxmlsetups xml:text \xmlflush{#1}\par \stopxmlsetups \starttext \xmlprocessbuffer{main}{test}{} \stoptext i still wonder what happens in tuc files (roundtrip mess) beta coming Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------

On 1/18/2016 10:45 PM, Thomas A. Schmitz wrote:

On 01/18/2016 10:26 PM, Hans Hagen wrote:

...

i already feared that ...

...

...

i still wonder what happens in tuc files (roundtrip mess)

beta coming

Yep, this seems to be a somewhat difficult area. Was just preparing an example when yours arrived. The latest beta seems to be doing the right thing. FYI: my example was an entity "linebreak" which I wanted to induce a linebreak (duh!) in the pdf but not when I serve the file via xsl...

aha. grep for "injector" in the test suite ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------