Hi, I was thinking about some utf-8 issues and where utf-8 might crop up. Please note that these are just musings at the moment and not a proposal. I am trying to get a hang of the situation and what kind of code would lead to what kind of complexity. Also, some ideas in here depend on a separate patch I am working on for PDFTeX (not yet completed). So this is _not_ intended as a work proposal or anything like that. It is just asking whether I have forgotten some aspects. It might lead to some proposals at a later point of time, but currently I am just trying to wrap my own head about some things. One thing I was thinking about was whether it might be a useful idea to make "char" an utf-8 entity (consisting of 4 bytes, either padded with 0 or with ff), so that ord and chr are not trivial operations. Also, a "packed array of char" would then want specific accessor functions. It would also be possible to represent unpacked arrays of char (and single characters) as UCS-32, but have packed arrays be coded in utf-8. Basically, the idea would be to move most of the utf-8/unicode issues to web2c, and then use something like iterators (defined in web2c with a suitable syntax) for accessing the input buffer. However, every place that currently has to explicitly deal with utf-8 (as compared to 8-bit TeX) would _still_ need to explicitly deal with utf-8 and would merely get different (and possibly more compact/natural) ways of expressing these kinds of conversions. So the benefits in "code pieces to touch" would not necessarily be impressive. So to get a better feeling about whether messing with web2c might be worth the trouble, I tried coming up with an estimate where utf-8 is actually used and/or useful, and in what ways. One place obviously is the input buffer. I am trying to find the time to work on the "partial line buffer" patch I talked about, where the necessity to read a whole line is no longer present. _If_ such a patch finds its way into PDFTeX, the space savings of utf-8 over UCS-32 will be irrelevant in the input buffer. UCS-32 would have the advantage of being able to more or less keep all the code like it is in PDFTeX. Without such a patch, however, the most compact input line buffer representation will appear desirable (as recent discussions about further extending the size clearly indicate). So what about the further workflow? From the input buffer, we have basically two uses: getting single characters (like in character nodes and similar): this would seem to imply UCS-32 as most efficient, and it is probably the most frequent operation all in all. Another use is looking up control sequence names. Those are stored in the string pool, and I think that utf-8 makes perfect sense there (mostly ASCII, possibly large amounts of data). Currently TeX uses the input buffer for interpreting control sequence names either directly from \... or from \csname ...\endcsname, so it appears necessary to put the input buffer into utf-8 if that is the coding in the string pool we want to compare with. However, one has to be aware that \csname ...\endcsname works from tokens, so its content is basically UCS-32 at the start. Now \... is certainly quite more common and important. I might, however, decide for the "partial line buffer" patch to use a _separate_ buffer for assembling control sequence names. Part of the reason is that TeX rearranges its input line when processing things like \r^^65lax with the result that error messages output lines that never occured in the source code. Treating the input lines in the buffer not as read-only data and already occasionally doing what amounts to "recoding" is an "optimization" that is really messy and probably not really worth the savings: control sequences, while occuring quite frequently in the input, will most likely be complex enough that the time needed for executing the control sequence is quite larger than the time needed for scanning it, and copying them into a separate place before looking them up is not likely to incur much overhead. So in the course of the partial line buffer patch to PDFTeX, I might introduce a separate place for assembly of control sequence names (which would in LuaTeX then be the suitable point for reconversion to utf-8), and the necessity for having control sequences and the input buffer coded in the same manner might then mostly disappear. Whether one makes use of it depends on where one would prefer to have the complexity: in the buffer handling code (which is distributed around TeX) or elsewhere. Not having to reconvert material from \csname...\endcsname and \scantokens and similar to utf-8 might also help to keep the utf-8 code complexity confined to few places. Where else is utf-8 interesting? One _very_ interesting place for it are hyphenation tables. It would seem to make a lot of sense to switch this sort of data structure into something utf-8 based as long as one makes sure that frequent prefix bytes (like C2) will not lead to inefficiency due to collisions. Since one can't specify breakpoints in _legal_ utf sequences _anywhere_ inside of a character's byte sequence, as long as the input is valid utf8, it seems like a perfectly usable idea to go via UTF-8. Disadvantage: since hyphenation is working from char nodes, this again implies having to convert UCS-32 to utf-8 before doing the hyphenation lookup. The current Aleph-inherited hyphenation works only in the Unicode baseplane and is 16-bit based and somewhat unstable (iniTeX will dump core when trying to hyphenate). Extending this to 21bit would either necessitate making use of surrogate pairs in UTF-16, or going to UTF-8. Since the complexities in either case are pretty much the same, UTF-8 would appear to be the more compact solution. One further consideration: \lefthyphenmin and \righthyphenmin will not be able to rely on the size of _byte_ strings. However, since they actually should rather count off grapheme clusters rather than single character codes (combining accents should not count as characters for \lefthyphenmin and \righthyphenmin, I think), one has to think about those separately, anyway. The hyphenation tables are special in that we would actually need to have utf-8 exposed as a _byte_ sequence with values from 0..255. Most other uses could be implemented with some packed array semantics inside of web2c, and using chr/ord. Did I forget places where utf-8 might or might not crop up? -- David Kastrup