Friday, December 2, 2005 Taco Hoekwater wrote:

Nikolai Weibull wrote:

...

We were planning on using Vim for preprocessing things like this instead of hacking TeX to do the lexing. I think Mojca (?) had some ideas for this and perhaps even a simple implementation, and given that there is now structured support for preprocessing in texexec I think that this can be done quite easily. A lot easier than writing something in TeX.

Speak for yourself. I'll take TeX over VIM anytime :-)

Eh, I'll have both TeX and Vim anytime :) Sheesh, even my IRC identity uses tex_vim as username and the UTF-8 equivalent of $\tau\epsilon\chi\nu\epsilon$ \&\ vis as "Real name" :) -- Giuseppe "Oblomov" Bilotta

Zeljko Vrba wrote:

On Fri, Dec 02, 2005 at 05:56:36PM +0100, Nikolai Weibull wrote:

...

We were planning on using Vim for preprocessing things like this instead

Vim for preprocessing? Isn't it a bit too painful compared to using a "real" programming language?

You misunderstand. Vim can easily be made to output the highlighting it would apply to a document on the display to something else, e.g., outputting TeX directives instead. A subthread on the subject can be found here: http://www.ntg.nl/pipermail/ntg-context/2005/012885.html It would require very little programming. syntax/2html.vim, which converts the buffer to a HTML document with syntax highlighting, is 526 lines in the current CVS incarnation. A syntax/2context.vim would be even shorter, perhaps 150 to 200 lines. If I find the time I’ll write something this weekend. I’m catching a could though, so I might not :-(. (Note: This would thus allow highlighted code for any format Vim understand, which is currently over 450!) nikolai -- Nikolai Weibull: now available free of charge at http://bitwi.se/! Born in Chicago, IL USA; currently residing in Gothenburg, Sweden. main(){printf(&linux["\021%six\012\0"],(linux)["have"]+"fun"-97);}

Nikolai Weibull wrote:

this than I have is welcome to finish it. The \highlight command should be defined something like this (pseudo-tex-code):

\pdef\highlight[#1]{#2}% {\bgroup \setupcolorforgroup[#1]% \type{#2}% \egroup}

#1 is a group name, such as Statement, Operator, or Comment. #2 may contain multiple lines, and I don’t know how well this will work on the TeX side. It may also contain special characters like {, #, &, and so on. Suggestions?

The most simple one I expect to work would be \def\highlight[#1]{% \bgroup \setuphighlightcolors[{#1}]% never forget those {}, % % since [] aren't balanced! \processinlineverbatim\egroup}% \egroup is an argument to % % \processinlineverbatim here (I did have some mildly elaborate low-level code here, but upon searching for the ConTeXt way of switching of everything, I came upon \processinlineverbatim, which probably does everything you want. If it does not, please come back and I'll send you something more low-level that should.) regards, Christopher

Christopher Creutzig wrote:

Nikolai Weibull wrote:

...

this than I have is welcome to finish it. The \highlight command should be defined something like this (pseudo-tex-code):

\pdef\highlight[#1]{#2}% {\bgroup \setupcolorforgroup[#1]% \type{#2}% \egroup}

#1 is a group name, such as Statement, Operator, or Comment. #2 may contain multiple lines, and I don�t know how well this will work on the TeX side. It may also contain special characters like {, #, &, and so on. Suggestions?

The most simple one I expect to work would be

\def\highlight[#1]{% \bgroup \setuphighlightcolors[{#1}]% never forget those {}, % % since [] aren't balanced! \processinlineverbatim\egroup}% \egroup is an argument to % % \processinlineverbatim here

(I did have some mildly elaborate low-level code here, but upon searching for the ConTeXt way of switching of everything, I came upon \processinlineverbatim, which probably does everything you want. If it does not, please come back and I'll send you something more low-level that should.)

there is no need for verbatim mode when you convert anyway! Hans

Nikolai Weibull wrote:

Nikolai Weibull wrote:

...

It would require very little programming. syntax/2html.vim, which converts the buffer to a HTML document with syntax highlighting, is 526 lines in the current CVS incarnation. A syntax/2context.vim would be even shorter, perhaps 150 to 200 lines. If I find the time I’ll write something this weekend. I’m catching a could though, so I might not :-(.

A splitting headache notwithstanding, here’s a syntax/2context.vim that weighs in at 170 lines. There are still things to do, like figuring out how to complement this on the ConTeXt side (i.e., defining \highlight) and things will depend on how this is done. Some sort of \type environment would be nice, as it is better to not do escaping of special characters on the Vim side. Someone with better knowledge of how to do this than I have is welcome to finish it. The \highlight command should be defined something like this (pseudo-tex-code):

\pdef\highlight[#1]{#2}% {\bgroup \setupcolorforgroup[#1]% \type{#2}% \egroup}

#1 is a group name, such as Statement, Operator, or Comment. #2 may contain multiple lines, and I don’t know how well this will work on the TeX side. It may also contain special characters like {, #, &, and so on. Suggestions?

don't use \type, just escape the characters \{ \} etc or use symbolic names and use \tttf (or \tt for tje whole code fragment; \type is overkill here an dmy do more hard than good so you get something \startVIMhightlighting \vhl[#1]{....} \stopVIMhighlighting with \def\startVIMhightlighting {\startlines \tt} \def\stopVIMhighlighting {\stoplines} \def\vhl[#1]#2% {\color[#1]{#2}} % maybe even local colors when speed is needed Even better, be more verbose: \startVIMhightlighting \vline{\vword[#1]{....}} \stopVIMhighlighting or even: \startVIMparagraph \startVIMline\VIMword[#1]{....} ... \stopVIMline \stopVIMparagraph in that case you have way more control (line numbering/treatment, spacing etc) [it's what i did with the scite exporter -- i output highligting in a compact XML where even spaces are tagged] Hans Hans Hans

Nikolai Weibull wrote:

Hans Hagen wrote:

[good tips on how to implement the TeX side of things]

Great, thanks!

...

[it's what i did with the scite exporter -- i output highligting in a compact XML where even spaces are tagged]

Is there (going to be) any support for this export format in ConTeXt so that I can just generate this kind of XML instead so that we only do things once? I found this in mscite-p.pdf: “The exporter will be descibed as soon as there are styles for processing the XML code. For the moment we stick to showing the schema.” Am I to take it that there are plans for handling this, but that nothing has been written yet? I’ll gladly help out if I can. I was hoping to make an article out of this, but now it seems that a lot of groundwork has already been done. (I really need to write some articles if I’m going to be accepted for a postgraduate position.)

see attached zip this is what is produced, writing an xml mapping to context code (i must have an example somewhere but cannot find it now) about an article ... it would be interesting to combine thsi with a kind of literate programming (if cweb would produce better output then it may have become a bit more popular) Hans

Nikolai Weibull wrote:

That’s just horrendous. The ‘n’ attribute on ‘line’ is useless (and why are empty lines unnumbered?). You can just as easily use XSLT’s <number/> to get that effect:

if i remember right it has to to with the fact that while editing inserted lines were treated different (sub numbers and such) so n is not om forehand something going up in steps of 1; it's more a status thing

Furthermore, the ‘n’ attribute on ‘t’ is equally useless. How am I to know what “3” stands for when converting?

in scite that's related to styles which have numbers (no symbolic names)

Finally, what’s the reason for having ‘s’? It just complicates matters immensely:

having access to spaces (without the need to xslt the file) can be handy if you want to treat them special (e.g. visualize them); keep in mind that in xml mode we don't have active character trickery available actually, it's one of the reasons why i dislike cdata being used for verbose content instead of tagged content <lines> <line>...</line> </lines> is easier to interface to certain typesetting things btw, when we process xml here, i often have to preprocess the files (using ruby and regexps) just in order to add some detail that permits proper typesetting

xsl:text

The problem is that the ‘n’ attribute has a default of 1, but as there’s no way to express default values for attributes in Relax-NG (and there’s no schema at the bogus url http://www.scintila.org/scite.rng) we’d have to hack it even more.

Could you enlighten me on the merits of having a tag for representing (sequences of) whitespace?

as said, in order to be able to treat them kind of special btw, it's all configurable: export.xml.collapse.spaces=1 export.xml.collapse.lines=1 so .. it's just what i needed (at that time) which was an alternative for the latex output mode that i could not use

Here’s a suggestion for a better grammar:

ID.datatype = xsd:ID LanguageCode.datatype = xsd:language

id.attrib = attribute id { ID.datatype }? xmlbase.attrib = attribute xml:base { text }? Core.attrib = id.attrib, xmlbase.attrib lang.attrib = attribute xml:lang { LanguageCode.datatype }? I18n.attrib = lang.attrib Common.attrib = Core.attrib, I18n.attrib

start = file | line | segment | whitespace

file = element file { file.attlist, file.content } file.attlist = Common.attrib, attribute path { text }, attribute type { text } file.content = line*

line = element line { line.attlist, line.content } line.attlist = Common.attrib line.content = (segment | whitespace)*

segment = element segment { segment.attlist, text } segment.attlist = Common.attrib, attribute type { text }

whitespace = element whitespace { whitespace.attlist, empty } whitespace.attlist = Common.attrib

I’m sure that there are things missing and that it can be improved further, but it’s a good start in my opinion.

What I can’t decide is whether to write things like

<line> <segment type="type">int</segment><whitespace/><segment type="normal">a;</segment> </line>

or

<line> <type>int</type><whitespace/><normal>a;</normal> </line>

The first scheme is easier to extend with more types, if we don’t validate the value of segment’s ‘type’ attribute (if we do, then both schemes are equally “easy” to extend), but the second scheme has the benefit that as much validation as possible can go into the schema and can thus ease translation.

I envision that the XSLT ConTeXt-converter for this scheme would have code in one of these two formats:

\highlight[]{}

or

\highlight[]{}

I really don’t know what I prefer. The first means that only the source and destination need to worry about dealing with new types (the stylesheet can be used unchanged).

it depends a bit on what an editor can provide; the scite xml output is rather connected to its lexer, and since each lexer is different there is no strict relationship between a number and e.g. a type. As a result, stylesheets need to have that kind of (conversion) knowledge. it's not clear to me how you want to use these code snippets - if you're going to copy them into your tex document, i'd leave them in xml format, i.e. have a mixed tex/xml doc (no need for conversion since we're dealing with rather simple xml->tex mapping sthat can be done by context itself) - if you keep them in the source file, you need a way to filter them into your main text document (this is what i do when i nedd to document for stylesheets and such: i filter the pieces i needed while typesetting)

...

about an article ... it would be interesting to combine thsi with a kind of literate programming

Yes, that’s true. I used a simple hack to my code written in literate programming into my master’s thesis. It worked quite well actually. I’m sure that this could be combined with the method I used there.

so how does this work? some text [reference to code snippet] some more text where you merge in the code snippets before processing?

...

(if cweb would produce better output then it may have become a bit more popular)

Yes. I see three problems with CWEB:

1. The syntax is too complicated 2. The output is not great 3. The input isn’t compilable in the source programming language

What my literate programming system did was allow for embedding comments in the source instead of the other way around (much like how ConTeXt sources are documented). This worked very well actually, especially in the parts of my code that were written in Ruby where one can do things like

# ¶ OK, so now that we have all the other methods done, it’s time to # write our final method \Ruby{method}. It will simply tie together the # other methods found in \Ruby{Class}:

why not # write our final method \RubyM[Class.method]. It will simply tie together the # other methods found in \RubyC[Class.method]: and let that info be auto-inserted? Now you have to edit three (probably more) places when you change the name of the method.

def Class.method ⋮ end

(A sequence of comments where the first comment began with a pilcrow sign are processed as stuff to send to ConTeXt and the following code block (basically everything up to the next pilcrow-endowned comment) was set inside a verbatim environment.)

Anyway, I’ve attached the Relax-NG schema that I deviced. Tomorrow I’m going to work on writing an exporter that exports to this format and a stylesheet that transforms this to ConTeXt code. If I have time I’ll also try to set up some environment for this and some generic preprocessing directives for texexec.

ok Hans