David Kastrup wrote:
I'll also try thinking about some input encoding implementation place: the process_input_buffer callback has several drawbacks: for one
You should not use that callback for input reencoding anyway (I thought you just did that because you wanted a quick hack). The idea is to use the 'open_read_file' callback instead, because that callback remains tied to the actual file. If you come up with a different system altogether, that is fine. But I should warn you: I will not even look at partial solutions. Either you fix reencoding for all cases, or not at all. And the same is true for buffer size limit problems. You have to figure out a way to with \scantokens, or you may as well forget about writing any code at all. I see absolutely no point in solving buffer overflows for files when we keep getting unrecoverable errors for \scantokens and luatex's tex.print(). Best wishes, Taco