Taco:
Attached is a provisional excerpt of the coming book on LuaTeX. It is ConTeXt source, but it should not be hard to get it to compile under LaTeX or plain if you don't have ConTeXt installed.
I've ConTeXt installed (in fact, the full TeX Live 2005). So, no problem.
From the excerpt:
From now on, whenever \LUATEX\ has to open a text file, it will call the function \type{file_opener} instead of actually opening the file itself. It stores the returned table in its memory, and it uses the function attached to the \type{reader} label for reading lines.
If I've understood correctly, this applies to files not yet opened, but usually the encoding is stated inside the file (ie, the file is already open).
function latin_to_utf (line) local s = ""; for c in string.bytes(line) do s = s .. unicode.utf8.char(c) end return s end
But where is the input encoding? Apparently this changes the "Unicode" representation from 8 bits (thus limited to the range 0-255, which is certainly latin-1) to utf-8, without reencoding anything (say, iso greek, koi8, macos, jis, etc.). I've googled for docs on unicode for lua but I haven't found anything particularly useful. Javier