[NTG-context] non-ascii chars in cmd.exe (Windows)

[Hans Hagen]

On 11/13/2020 2:49 PM, Pablo Rodriguez wrote:
> On 11/12/20 10:55 PM, Hans Hagen wrote:
>> On 11/12/2020 10:39 PM, Pablo Rodriguez wrote:
>>> Dear list,
>>>
>>> I have the following sample:
>>>
>>>     \starttext
>>>     \startluacode
>>>     io.write(' Name? ')
>>>     document.name = io.read() or ''
>>>     \stopluacode
>>>     \cldcontext{document.name} is the name.
>>>     \stoptext
>>> [...]
>>> I’m using current latest (ConTeXt MkIV 2020.11.08 12:42).
>>>
>>> I’m not sure what I am missing or whether I have hit a bug.
>>
>> This has been discussed before I think. It has to do with how you
>> configured your system (what encoding) and how consisteltly you take
>> that into account. If you have some mixed setup, just don't use non-ascii.
> 
> Many thanks for your reply, Hans.
> 
> I use chcp 65001. I have no problem displaying UTF-8 messages and
> passing arguments with UTF-8 characters.
> 
>> Anyway, in LMTX all file, commandline and systemn operations are utf 8
>> and on windows get translated into wide system calls so there it should
>>    work ok if you use utf8.
> 
> Well, according to https://ss64.com/nt/cmd.html the console uses
> UTF-16LE. Or UCS-2 (as described in
> https://devblogs.microsoft.com/commandline/windows-command-line-unicode-and-utf-8-output-text-buffer/#console-built-in-a-pre-unicode-dawn).
> 
> Since console input is UTF-16LE, I guess I may need something in Lua
> similar to 'regimes.translate(str, "utf16le")'.
the console uses whatever code page you have configured and it also 
relates to the code page used for filenames .. the 16 bit values are 
used deep down and what you use gets translated into that (often folks 
still use some 8 bit code page so that then gets mapped) .. there is no 
way the system can know if what you provide is as it's bytes in whatever 
encoding used

so, you need to look what your system is configured for

(this is independent of the output to the console which is what the 
65001 does)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------