[NTG-context] ConTeXt Minimals parsing of .tui broken with ruby 1.9.1 in Windows

13 Jul 2009

      Hello all,

I want to report a problem that is either in ConTeXt, or in ruby 1.9.1
(last version of ruby). More probably, the problem has to do with ruby
handling non-ASCII characters. I have no means of trying Linux, Solaris,
etc...
Anyone using ConTeXt with ruby 1.9.1 will face it probably (at least in
Windows :-)

The problem happens with all files, even with the simple "Hello":

    \starttext
    Hello World
    \stoptext

After installing ConTeXt Minimals (the devel version) yesterday,
I ran the above example with ruby 1.9.1-p129 in Windows
(both Win 2000 and XP show the problem).

Meanwhile I compiled and tried several versions of Ruby, and found the
following pattern of problems:

ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32]     PROBLEM
ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-mingw32]        PROBLEM
ruby 1.9.0 (2008-10-04 revision 19669) [i386-mingw32]          No problem
ruby 1.8.7 (2009-06-12 patchlevel 174) [i386-mingw32]          No problem
ruby 1.8.6 (2009-06-08 patchlevel 369) [i386-mingw32]          No problem

So, whatever it is, it is broken with ruby 1.9.1.
All the versions of ruby were compiled in Windows using the mingw toolchain,
with GCC 3.4.5.
...
...
...
...
Here is the description of what happens   <<<<
When texutil parses the .tui file, I get the following (see comments after
this text output):

---------------------------------------------
    ...........................
    Output written on con-hello1.pdf (1 page, 21759 bytes).
    Transcript written on con-hello1.log.
    TeXUtil | parsing file con-hello1.tui
    TeXUtil | debug 1 jasa #File:0x1271d18
    xxx c \thisissectionseparator{-}
    xxx c \thisisutilityversion{2008.10.14}
    xxx c \thisisbytesequence{?+Ç}
    TeXUtil | fatal error in parsing con-hello1.tui
    TeXUtil | shortcuts : 0
    TeXUtil | expansions: 0
    TeXUtil | reductions: 0
    TeXUtil | divisions : 0
    TeXUtil | loaded files: 0
    TeXUtil | temporary files: 0
    TeXUtil | commands: 2
    TeXUtil | programs: 0
    TeXUtil | tuo file saved
    TeXExec | runtime: 2.703125
--------------------------------------------

The lines with "debug 1 jasa" and starting with "xxx" result from
the simple "debug code" I inserted in the file texutil.rb to find the
problematic line. The error happens when the following ruby code
is executed: (the extra  debug lines have a mark # jasa )

---------------------- texutil.rb (snippet, around line 1025)
----------------------

        def loaded(filename)
            begin
                tuifile = File.suffixed(filename,'tui')
                if FileTest.file?(tuifile) then
                    report("parsing file #{tuifile}")
                    if f = open(tuifile) then
            report("debug 1 jasa #{f}")  # jasa
                        f.each do |line|
                print "xxx #{line}"  # jasa
                            case line.chomp
                                when /^f (.*)$/o then
@plugins.reader('MyFiles',    $1.splitdata)
                                when /^c (.*)$/o then
@plugins.reader('MyCommands', [$1])
                                when /^e (.*)$/o then
@plugins.reader('MyExtras',   $1.splitdata)
                                when /^s (.*)$/o then
@plugins.reader('MySynonyms', $1.splitdata)
                                when /^r (.*)$/o then
@plugins.reader('MyRegisters',$1.splitdata)
                                when /^p (.*)$/o then
@plugins.reader('MyPlugins',  $1.splitdata)
                                when /^x (.*)$/o then
@plugins.reader('MyKeys',     $1.splitdata)
                                when /^r (.*)$/o then # nothing, not handled
here
                            else
                                # report("unknown entry #{line[0,1]} in line
#{line.chomp}")
                            end
                end
                        f.close
                    end
                else
                    report("unable to locate #{tuifile}")
                end
            rescue
                report("fatal error in parsing #{tuifile}")
                @filename = 'texutil'
            else
                @filename = filename
            end
        end

---------------------------------------------------------------
...
From the debugging lines that are expelled, it is clear that the line in the
.tui file that triggers the problem is:
c \thisisbytesequence{ ...non-ASCII codes... }

and, precisely, it s the second line of the 'case':

    when /^c (.*)$/o then @plugins.reader('MyCommands', [$1])

which processes the .tui line and triggers the 'rescue' clause. So I think
the problem lies in the "digestion"
of non-ASCII characters by the last version of Ruby.

I don't know what is the meaning of the \thisisbytesequence line in ConTeXt
and the maening of those non-ASCII chars. I followed the
@plugins.reader('MyCommands', [$1])
and figured out that what raises the exception happens before the
@plugins.reader method,
since it is never reached when the  "\thisisbytesequence" line is processed.

To finish, I think this can be of interest for Ruby guys, but is also
breaking ConTeXt for anyone with ruby 1.9.1 installed in Windows.

Thanks,

Jose Soares Augusto

jasaugusto@gmail.com
jasa@fisica.fc.ul.pt

[NTG-context] ConTeXt Minimals parsing of .tui broken with ruby 1.9.1 in Windows

Jose Augusto