On Sat, 05 Jun 2004 22:41:39 +0200, Thomas A. Schmitz wrote:

Idris,

I know a bit of perl and would love to help. However, I fear that sending us your stuff via mail will be a bit difficult because the utf-8 chracters get transformed into gibberish.

Thnx 4 such a speedy reply! I don't think you are getting gibberish though; you should be getting the extended ascii representation. So the letter alif (hex 0627) should look like this: ا Do you get a forward-slashed circle and a section symbol? If so, that's the ascii representation I'm trying to convert to the letter `A'. Here are the codes you want: ا [0627] => A ب [0628] => b ج [062C] => j د [062F] => d ه [0647] => h و [0648] => w ز [0632] => z Let me explain my situation more clearly:-) I have a unicode editor, Unitype Global Writer. I save a unicode document as a utf *.txt file. When I open that saved file in my TeX editor (WinEdt), it comes out as extended ascii (that's the "gibberish"). So what I wanted to do was convert the ascii "gibberish" to my Latin transcription. It seems that what you are suggesting is to use the hex representation and convert the unicode txt file into a Latin transcription file directly and bypass the gibberish. On your perl file, can you give me an example of how to use it? I tried (in windows, with name utf2tex.pl and unicode text in unicode-utf.txt) and get =========================

perl utf2tex.pl unicode-utf.txt Unknown discipline class ':utf8' at C:/Perl/lib/open.pm line 18. BEGIN failed--compilation aborted at utf2tex.pl line 4. =========================

from your script I tried, e.g. ============================ $_ =~ s/\x{0627}/\x{0041}/esg; # from alif to `A' ============================ Your guidance will be greatly appreciated! Thnx a million! Idris -- Professor Idris Samawi Hamid Department of Philosophy Colorado State University Fort Collins, CO 80523

On Sat, 05 Jun 2004 23:48:18 +0200, Thomas A. Schmitz wrote:

Just a quick reply (it's bedtime over here): there may be 2 problems.

Ok, get some sleep;-) Anyhow, I fixed the line break (is the space between tilda and `s' correct?) ============================== $_ =~ s/\x{0627}/\x{0041}/esg; #this is the actual conversion ============================== did not work though:-( My perl version is v5.6.1; I went to the ActivePerl website and the only version they had is v5.6.1.638; so from perl.org I found Indigoperl and switched;-) This solves part of the problem:-) Now I get

perl utf2tex.pl unicode-utf.txt syntax error at utf2tex.pl line 8, near ");" Execution of utf2tex.pl aborted due to compilation errors.

line 8 is while (<>); { #this opens the file for reading Here is the whole file once again: ================================== #!/usr/bin/perl -w use strict; use open ':utf8'; open(NEW,">new.tex"); #opens file to print out the result while (<>); { #this opens the file for reading $_ =~ s/\x{0627}/\x{0041}/esg; #this is the actual conversion print NEW "$_"; #and this writes the result into file "new.tex" } close(NEW); ==================================

the usage is right (I'm not a windows person,

If WinEdt (and some other things) worked under WINE, I would not be a windows person either:-( And will attempt yet another switch (I've lost count) to Linux-KDE sometime this Summer... Thnx a million Idris -- Professor Idris Samawi Hamid Department of Philosophy Colorado State University Fort Collins, CO 80523

On Sun, 6 Jun 2004 01:15:56 +0200, Giuseppe Bilotta wrote:

My take: try the following (should work even with ActiveState 5.6)

=== #!/usr/bin/perl

use strict; #D comment the following, I think we can do without # use open ':utf8';

open(NEW,">new.tex"); #opens file to print out the result

while (<>); { #this opens the file for reading

$_ =~ s/\x06\x27/A/esg; #this is the actual conversion

print NEW "$_"; #and this writes the result into file "new.tex" }

close(NEW); ===

Hi Giuseppe (Is it not way past your bedtime;->), Here's my result:

perl utf2tex2.pl < unicode-utf.txt syntax error at utf2tex2.pl line 9, near ");" Bareword "A" not allowed while "strict subs" in use at utf2tex2.pl line 11. Execution of utf2tex2.pl aborted due to compilation errors.

please advise;-> best Idris -- Professor Idris Samawi Hamid Department of Philosophy Colorado State University Fort Collins, CO 80523

On Sun, 6 Jun 2004 01:58:44 +0200, Giuseppe Bilotta wrote:

== #!/usr/bin/perl

use strict; use warnings;

open(NEW,">new.tex"); #opens file to print out the result

while (<>) { #this opens the file for reading

$_ =~ s/\xD8\xA7/A/g; #this is the actual conversion $_ =~ s/\xD8\xA8/b/g; #this is the actual conversion $_ =~ s/\xD8\xAC/j/g; #this is the actual conversion $_ =~ s/\xD8\xAF/d/g; #this is the actual conversion $_ =~ s/\xD9\x87/h/g; #this is the actual conversion $_ =~ s/\xD9\x88/w/g; #this is the actual conversion $_ =~ s/\xD8\xB2/z/g; #this is the actual conversion

print NEW "$_"; #and this writes the result into file "new.tex" }

close(NEW); ===

It works! I'll try to finish a basic script that works for Lagally's ArabTeX transcription (that I use) and post it here and on the aleph list. One question: The hex for e.g. alif is 0627; how did you get D8A7 from that for purposes of the script (so I can follow along for the rest)? Best Idris -- Professor Idris Samawi Hamid Department of Philosophy Colorado State University Fort Collins, CO 80523

Am Sonntag, 06.06.04, um 02:19 Uhr (Europe/Zurich) schrieb Idris Samawi Hamid:

...

open(NEW,">new.tex"); #opens file to print out the result

better: open NEW, ">", "new.tex" || die $!;

...

$_ =~ s/\xD8\xA7/A/g; #this is the actual conversion

if you work with $_ you can leave it out, simply: s/\xD8\xA7/A/g; But for a series of conversions I'd suggest an hash for better overview. Whole script like this: ----- #!/usr/bin/perl -w use strict; use warnings; my ($Source, $Target) = (shift, shift); # gets 2 file names from command line my %conv = ( # enhance as needed "\xD8xA7" => "A", "\xD8xA8" => "b", "\xD8xAC" => "j", "\xD8xAF" => "d" ); open SOURCE, "<", $Source || die $!; open TARGET, ">", $Target || die $!; # there are ways to read a whole file in one scalar, # e.g. with File::Slurp, but I don't know them by heart... while (my $line = <SOURCE>) { foreach my $key (keys %conv) { $line =~ s/$key/$conv{$key}/g; } # foreach print TARGET $line; } # while close SOURCE; close TARGET; ----- BTW: ActiveState has Perl 5.8.4, at least for Windows (I use it at work). Grüßlis vom Hraban! -- http://www.fiee.net/texnique/

Well, if you put the use open ':utf8'; in the header of your perl script, it should work without the hex editor (btw: I would recommend using emacs in hex mode (M-x hexl-find-file). And just for the record: to put the entire file in one array, use this: my @lines = <>; my $text = join "", @lines; $text =~ s/PUT_YOUR/SUBSTITUIONS_HERE/esg; But it looks like you got a working solution now, so have fun playing around with it. And boy does it make one feel good when you realize that you windoze people are still working with perl 5.6 -- that's the stone age, man ;-) Best Thomas On Sun, 2004-06-06 at 23:03, Idris Samawi Hamid wrote:

On Sun, 6 Jun 2004 11:09:32 +0200, Henning Hraban Ramm wrote:

...

-----

#!/usr/bin/perl -w use strict; use warnings;

my ($Source, $Target) = (shift, shift); # gets 2 file names from command line

my %conv = ( # enhance as needed "\xD8xA7" => "A", "\xD8xA8" => "b", "\xD8xAC" => "j", "\xD8xAF" => "d" );

open SOURCE, "<", $Source || die $!; open TARGET, ">", $Target || die $!; # there are ways to read a whole file in one scalar, # e.g. with File::Slurp, but I don't know them by heart... while (my $line = <SOURCE>) { foreach my $key (keys %conv) { $line =~ s/$key/$conv{$key}/g; } # foreach print TARGET $line; } # while close SOURCE; close TARGET;

-----

Thnx; I'll play around with this as well. BTW: is there any way to do this without the hex editor and just enter the full 4-digit character (a la Thomas's original suggestion) e.g.,

"\x0627" => "A"

While the hex editor certainly works it is really slow and tedious work...

...

BTW: ActiveState has Perl 5.8.4, at least for Windows (I use it at work).

Ok, I found it:

http://downloads.activestate.com/ActivePerl/Windows/5.8/ActivePerl-5.8.3.809...

But the web site (at first glance) sure gives one the impression that their latest release is 5.6.1.638

http://www.activestate.com/

http://www.activestate.com/Products/ActivePerl/

Best Idris

Hey Hraban, not to be a PITA, but with OS X 10.3, we moved up to perl 5.8.1. And in my gentoo installation, I'm now at perl 5.8.4, yessirre! ;-) Best Thomas On Jun 7, 2004, at 9:45 PM, Henning Hraban Ramm wrote:

MacOS X has also only 5.6 if you don't install a newer one yourself, and with newer than 5.8.0 you get endless trouble...

On Sun, 6 Jun 2004 11:08:46 +1200, Richard MAHONEY wrote:

You might like to look at some of the encoding conversion scripts at:

http://homepages.comnet.co.nz/~r-mahoney/scripts/scripts.html

N.B. For sorting utf-8 Arabic you might find the perl module `Sort::ArbBiLex' useful

Thnx Richard; most of the scripts are beyond my capability;->, but the UTF8 to TeX / LaTeX scripts seem very useful for a possible transliteration module, which is something else I need to do for my journal, where people send in different transliteration conventions that I have to convert to the journal's convention. I'm ashamed to say it; I've been doing this kind of thing manually up to now; \startsuperhero must... learn... scripting... language... \stopsuperhero Thnx Idris -- Professor Idris Samawi Hamid Department of Philosophy Colorado State University Fort Collins, CO 80523