[Dev-luatex] UTF-16 in \pdfoutline

[Heiko Oberdiek]

On Tue, Dec 11, 2007 at 09:07:31AM +0100, Jonathan Sauer wrote:

> % The following code converts a string to UTF-16 big endian with BOM
> % and outputs it using \message:

> 	function convertToUTF16(str)
> 		local result = string.char(0xFE) .. string.char(0xFF)
> 		for c in string.utfvalues(str) do
> 			if c < 0x10000 then
> 				result = result ..
> 						 string.char(c / 256) ..
> 						 string.char(c % 256)
> 			else

> 		tex.print('\\message{' .. result .. '}')

> This fails with 'Text line contains an invalid utf-8 sequence.' (not
> surprising, since the text is UTF-16 big endian). If I want to pass the
> UTF-16-encoded string i.e. to \pdfoutline (since PDF bookmarks can be
> encoded in UTF-16), how do I do this?

In package "pdftexcmds" I had a similar problem with the reprogramming
of \pdfunescapehex. Here also 8bit bytes are possible as result.
My solution was:
* Use of a token register (avoids catcode troubles), principle:
    \newtoks\foobartoks
    \directlua0{
        ...
        function convert(str)
          ...
          tex.settoks("foobartoks", <result>)
          ...
        end
    }
    \def\convert#1{%
      \the\directlua0{%
        convert("\luaescapestring{#1}")
      }\foobartoks
    }
  Alternatively tex.print may be used, but it requires a catcode table
  to avoid trouble with unexpected catcode settings. (Currently
  I am writing a package for this purpose.)
* The convert function in Lua first calculates the byte string.
  Then it have replaced the bytes in the lua string by its
  multi-byte sequence in UTF-8. Then LuaTeX will see proper
  UTF-8 input and converts it back.

Of course, I want to add support for hyperref. But currently I don't
know, what a char token between 128 and 256 means:
* Proper Unicode?
* A byte of other encoding?
* Meaning of catcode (11, 12, 13)?

Yours sincerely
  Heiko <oberdiek at uni-freiburg.de>