
\def\LUATEX{Lua\TEX}

\starttext

\section {Finding files}

You can completely control what \LUATEX\ does when the user has
written a command like \type{\input plain}, by setting up a callback
function for \LUATEX\ to execute instead of the default action of
using kpathsea to both find and open the file.

There are basic two reasons why you may want to overrule this
behaviour. The first is that you are perhaps not happy with how
kpathsea finds files (perhaps because you want to read files from
non||standard locations), and the second that you may want to do
something special once the file has been found and is going to be read
(setting up the \type{'find_read_file'} callback is currently a
prerequisite for the use of \type{'open_read_file'} that will be
explained in the next section).

Without much ado, here is a the simplest way to set up a
callback for text file discovery:

\starttyping
function file_finder (number, fname)
   return kpse.find_file(fname)
end
callback.register('find_read_file', file_finder);
\stoptyping

The \type{number} argument is a number ranging from 0 to 16 that will
be explained later. The \type{fname} is a string representing the
user's input file name (\type{plain} in the above example).

This callback function does essentially the same thing as the default
behaviour, by calling the library function \type{kpse.find_file()}.

\section {File formats and encodings}

The bare \LUATEX\ executable only understands files in Unicode
encoding using UTF-8 transformation as a file format.

But that does not mean that you have to convert all your files to
UTF-8 before running \LUATEX: there are a couple of possible places
where you can change the input before the executable actually looks at
it.

The first of these places is right in the middle of the file being
read. On top of defining a callback to find a file as explained in the
previous section, you can also set up a callback for the opening of a
text file, like so:

\starttyping
function file_opener (fname)
   ... 
end
callback.register('open_read_file', file_opener);
\stoptyping

The \type{file_opener} function receives the full file name of the
file to be opened as it's argument, and should return a table with at
least a \type{reader} key whose value is a function that can be called
by \LUATEX.  It will be called repeatedly to retrieve the next line
from the file, until it returns \type{nil}, indicating end||of||file. 

A really simple version of \type{file_opener} could done like so: 

\starttyping
function file_opener (fname)
   local f = io.open(fname)
   if f == nil then 
      return nil
   else
      local readline = function ()  return f:read()  end 
      local t = { }
      t.reader = readline
      return t
   end
end
callback.register('open_read_file', file_opener);
\stoptyping

So, we have created the function \type{file_opener} that itself
defines the function \type{readline}, and gives this function back 
as the entry \type{reader} in the table \type{t}. 

>From now on, whenever \LUATEX\ has to open a text file, it will call
the function \type{file_opener} instead of actually opening the file
itself. It stores the returned table in its memory, and it uses the
function attached to the \type{reader} label for reading lines.

The task of the \type{reader} function is to return a UTF-8 encoded
string representing a line from a file, minus the line end symbol.  In
the above example, \type{readline} is really simple. This is suitable
if you know that the file is already in UTF-8 encoding: the Lua
function \type{f:read()} already returns the line minus line end
character(s) so nothing more needs doing.

Now, say that the file is not UTF-8 unicode, but latin-1 encoding.  To
convert latin-1 data into UTF-8 we have to call a function to convert
the line that we just read into the final version that we want to pass
on to \LUATEX. This of course requires that there {\it is\/} a line,
so we had better test for end||of||file. The conversion is better
handled in a separate function, so \type{readline} becomes

\starttyping
local readline = function ()
   local s = "";
   local line = f:read()
   if line == nil  then
      return nil
   else
     return latin_to_utf(line)
   end
end
\stoptyping

For conversion of a latin-1 string to utf-8, only a change in the
presentaion of the values is needed, and there is a handy function
that does precisely that.

The \type{unicode.utf8.char} function expects a byte as input, and the
\type{readline} funtion expects a string back, so a loop over the 
line is needed that dissects it into bytes and then reconstructs a
string.

The definition of the conversion function is:

\starttyping
function latin_to_utf (line)
   local s = "";
   for c in string.bytes(line) do
      s = s .. unicode.utf8.char(c)
   end
   return s
end
\stoptyping

Putting it all together gives:

\starttyping
function latin_to_utf (line)
   local s = "";
   for c in string.bytes(line) do
      s = s .. unicode.utf8.char(c)
   end
   return s
end
function file_opener (fname)
   local f = io.open(fname)
   if f == nil then 
      return nil
   else
      local readline = function ()
         local s = "";
         local line = f:read()
         if line == nil  then
           return nil
         else
           return latin_to_utf(line)
         end
      end
      return { reader = readline }
   end
end
callback.register('open_read_file', file_opener);
\stoptyping


\stoptext
