On Tue, Dec 11, 2007 at 09:07:31AM +0100, Jonathan Sauer wrote:
% The following code converts a string to UTF-16 big endian with BOM % and outputs it using \message:
function convertToUTF16(str) local result = string.char(0xFE) .. string.char(0xFF) for c in string.utfvalues(str) do if c < 0x10000 then result = result .. string.char(c / 256) .. string.char(c % 256) else
tex.print('\\message{' .. result .. '}')
This fails with 'Text line contains an invalid utf-8 sequence.' (not surprising, since the text is UTF-16 big endian). If I want to pass the UTF-16-encoded string i.e. to \pdfoutline (since PDF bookmarks can be encoded in UTF-16), how do I do this?
In package "pdftexcmds" I had a similar problem with the reprogramming
of \pdfunescapehex. Here also 8bit bytes are possible as result.
My solution was:
* Use of a token register (avoids catcode troubles), principle:
\newtoks\foobartoks
\directlua0{
...
function convert(str)
...
tex.settoks("foobartoks", <result>)
...
end
}
\def\convert#1{%
\the\directlua0{%
convert("\luaescapestring{#1}")
}\foobartoks
}
Alternatively tex.print may be used, but it requires a catcode table
to avoid trouble with unexpected catcode settings. (Currently
I am writing a package for this purpose.)
* The convert function in Lua first calculates the byte string.
Then it have replaced the bytes in the lua string by its
multi-byte sequence in UTF-8. Then LuaTeX will see proper
UTF-8 input and converts it back.
Of course, I want to add support for hyperref. But currently I don't
know, what a char token between 128 and 256 means:
* Proper Unicode?
* A byte of other encoding?
* Meaning of catcode (11, 12, 13)?
Yours sincerely
Heiko