ConTeXt Minimals parsing of .tui broken with ruby 1.9.1 in Windows
Hello all, I want to report a problem that is either in ConTeXt, or in ruby 1.9.1 (last version of ruby). More probably, the problem has to do with ruby handling non-ASCII characters. I have no means of trying Linux, Solaris, etc... Anyone using ConTeXt with ruby 1.9.1 will face it probably (at least in Windows :-) The problem happens with all files, even with the simple "Hello": \starttext Hello World \stoptext After installing ConTeXt Minimals (the devel version) yesterday, I ran the above example with ruby 1.9.1-p129 in Windows (both Win 2000 and XP show the problem). Meanwhile I compiled and tried several versions of Ruby, and found the following pattern of problems: ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32] PROBLEM ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-mingw32] PROBLEM ruby 1.9.0 (2008-10-04 revision 19669) [i386-mingw32] No problem ruby 1.8.7 (2009-06-12 patchlevel 174) [i386-mingw32] No problem ruby 1.8.6 (2009-06-08 patchlevel 369) [i386-mingw32] No problem So, whatever it is, it is broken with ruby 1.9.1. All the versions of ruby were compiled in Windows using the mingw toolchain, with GCC 3.4.5.
Here is the description of what happens <<<<
When texutil parses the .tui file, I get the following (see comments after this text output): --------------------------------------------- ........................... Output written on con-hello1.pdf (1 page, 21759 bytes). Transcript written on con-hello1.log. TeXUtil | parsing file con-hello1.tui TeXUtil | debug 1 jasa #File:0x1271d18 xxx c \thisissectionseparator{-} xxx c \thisisutilityversion{2008.10.14} xxx c \thisisbytesequence{?+Ç} TeXUtil | fatal error in parsing con-hello1.tui TeXUtil | shortcuts : 0 TeXUtil | expansions: 0 TeXUtil | reductions: 0 TeXUtil | divisions : 0 TeXUtil | loaded files: 0 TeXUtil | temporary files: 0 TeXUtil | commands: 2 TeXUtil | programs: 0 TeXUtil | tuo file saved TeXExec | runtime: 2.703125 -------------------------------------------- The lines with "debug 1 jasa" and starting with "xxx" result from the simple "debug code" I inserted in the file texutil.rb to find the problematic line. The error happens when the following ruby code is executed: (the extra debug lines have a mark # jasa ) ---------------------- texutil.rb (snippet, around line 1025) ---------------------- def loaded(filename) begin tuifile = File.suffixed(filename,'tui') if FileTest.file?(tuifile) then report("parsing file #{tuifile}") if f = open(tuifile) then report("debug 1 jasa #{f}") # jasa f.each do |line| print "xxx #{line}" # jasa case line.chomp when /^f (.*)$/o then @plugins.reader('MyFiles', $1.splitdata) when /^c (.*)$/o then @plugins.reader('MyCommands', [$1]) when /^e (.*)$/o then @plugins.reader('MyExtras', $1.splitdata) when /^s (.*)$/o then @plugins.reader('MySynonyms', $1.splitdata) when /^r (.*)$/o then @plugins.reader('MyRegisters',$1.splitdata) when /^p (.*)$/o then @plugins.reader('MyPlugins', $1.splitdata) when /^x (.*)$/o then @plugins.reader('MyKeys', $1.splitdata) when /^r (.*)$/o then # nothing, not handled here else # report("unknown entry #{line[0,1]} in line #{line.chomp}") end end f.close end else report("unable to locate #{tuifile}") end rescue report("fatal error in parsing #{tuifile}") @filename = 'texutil' else @filename = filename end end ---------------------------------------------------------------
From the debugging lines that are expelled, it is clear that the line in the .tui file that triggers the problem is:
c \thisisbytesequence{ ...non-ASCII codes... } and, precisely, it s the second line of the 'case': when /^c (.*)$/o then @plugins.reader('MyCommands', [$1]) which processes the .tui line and triggers the 'rescue' clause. So I think the problem lies in the "digestion" of non-ASCII characters by the last version of Ruby. I don't know what is the meaning of the \thisisbytesequence line in ConTeXt and the maening of those non-ASCII chars. I followed the @plugins.reader('MyCommands', [$1]) and figured out that what raises the exception happens before the @plugins.reader method, since it is never reached when the "\thisisbytesequence" line is processed. To finish, I think this can be of interest for Ruby guys, but is also breaking ConTeXt for anyone with ruby 1.9.1 installed in Windows. Thanks, Jose Soares Augusto jasaugusto@gmail.com jasa@fisica.fc.ul.pt
Jose Augusto wrote:
Hello all,
I want to report a problem that is either in ConTeXt, or in ruby 1.9.1 (last version of ruby). More probably, the problem has to do with ruby handling non-ASCII characters. I have no means of trying Linux, Solaris, etc... Anyone using ConTeXt with ruby 1.9.1 will face it probably (at least in Windows :-)
The problem happens with all files, even with the simple "Hello":
\starttext Hello World \stoptext
After installing ConTeXt Minimals (the devel version) yesterday, I ran the above example with ruby 1.9.1-p129 in Windows (both Win 2000 and XP show the problem).
(maybe mojca can patch this in core-uti.mkii: ): % \appendtoks % \immediatewriteutilitycommand{\thisisbytesequence{\testbytesequence}}% % \to \everyopenutilities \let\testbytesequence \empty % keep this \let\thisisbytesequence\gobbleoneargument % keep this The reason for this test is that in the past there were engines around that were 8 bit but configured to be 7 bit. Especially tetex was a problem but as that is now obsolete we can remove this test.
Meanwhile I compiled and tried several versions of Ruby, and found the following pattern of problems:
ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32] PROBLEM ruby 1.9.1p0 (2009-01-30 revision 21907) [i386-mingw32] PROBLEM ruby 1.9.0 (2008-10-04 revision 19669) [i386-mingw32] No problem ruby 1.8.7 (2009-06-12 patchlevel 174) [i386-mingw32] No problem ruby 1.8.6 (2009-06-08 patchlevel 369) [i386-mingw32] No problem
So, whatever it is, it is broken with ruby 1.9.1. All the versions of ruby were compiled in Windows using the mingw toolchain, with GCC 3.4.5.
actually, ruby 1.9+ broke the scripts anyway, as some file related functionality was no longer available (and moved to modules) so i already adapted all the ruby scripts
From the debugging lines that are expelled, it is clear that the line in the ..tui file that triggers the problem is:
c \thisisbytesequence{ ...non-ASCII codes... }
and, precisely, it s the second line of the 'case':
when /^c (.*)$/o then @plugins.reader('MyCommands', [$1])
which processes the .tui line and triggers the 'rescue' clause. So I think the problem lies in the "digestion" of non-ASCII characters by the last version of Ruby.
looks that way ... quite disturbing if they changed the default; maybe they move to utf8 but then i'd expect that to happen in 2+ versions thanks for looking into it Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com | www.pragma-pod.nl -----------------------------------------------------------------
After installing ConTeXt Minimals (the devel version) yesterday, I ran the above example with ruby 1.9.1-p129 in Windows (both Win 2000 and XP show the problem).
(maybe mojca can patch this in core-uti.mkii: ):
% \appendtoks % \immediatewriteutilitycommand{\thisisbytesequence{\testbytesequence}}% % \to \everyopenutilities
\let\testbytesequence \empty % keep this \let\thisisbytesequence\gobbleoneargument % keep this
Done, but untested. Mojca
Hi all, Thanks for the patch. I just updated ConTeXt Minimals and re-tried. Here is the GOOD result, now its working: ----------------------------------------- F:\ANOS\TeXes>ruby -v ruby 1.9.1p129 (2009-05-12 revision 23412) [i386-mingw32] F:\ANOS\TeXes>texexec con-hello1.tex TeXExec | processing document 'con-hello1.tex' .................................................................... Output written on con-hello1.pdf (1 page, 21759 bytes). Transcript written on con-hello1.log. TeXUtil | parsing file con-hello1.tui TeXUtil | shortcuts : 169 TeXUtil | expansions: 308 TeXUtil | reductions: 0 TeXUtil | divisions : 0 TeXUtil | loaded files: 1 TeXUtil | temporary files: 0 TeXUtil | commands: 20 TeXUtil | programs: 0 TeXUtil | tuo file saved TeXExec | runtime: 4.578125 ----------------------------------------------------------------------------- Meanwhile, the "evil line" with "ctrl chars" is not anymore in the .tui file. I want to thank Hans and Mojca for the patching and the kindness. Jose. On Tue, Jul 14, 2009 at 1:56 PM, Mojca Miklavec < mojca.miklavec.lists@gmail.com> wrote:
After installing ConTeXt Minimals (the devel version) yesterday, I ran the above example with ruby 1.9.1-p129 in Windows (both Win 2000 and XP show the problem).
(maybe mojca can patch this in core-uti.mkii: ):
% \appendtoks % \immediatewriteutilitycommand{\thisisbytesequence{\testbytesequence}}% % \to \everyopenutilities
\let\testbytesequence \empty % keep this \let\thisisbytesequence\gobbleoneargument % keep this
Done, but untested.
Mojca
___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://tex.aanhet.net archive : https://foundry.supelec.fr/projects/contextrev/ wiki : http://contextgarden.net
___________________________________________________________________________________
participants (3)
-
Hans Hagen -
Jose Augusto -
Mojca Miklavec