This layer is certainly needed for a program inteded to be used
worldwide on as much operating systems as possible (or on as many as
there are volunteers to port it to).
It's already required for Windows plus any Unix system (e.g. Windows + OS X).
This will work, I think, for OS that do not support UTF-8.
The only such OS is Windows.
utf8 -> w_char's (Provide some dummy solutión for values >2^16,
e.g. c & 0xFFFF)
Don't use the dummy solution; Windows uses UTF-16.
w_char's -> chars via wcstombs()
No, Windows uses UTF-16. This step is unnecessary and harmful.
wcstombs() is affected by the locale settings. According to the C
language specification
setlocale( LC_ALL, "" ); //Sets the locale to the native environment.
In Windows this is the user-default ANSI code page obtained from the
operating system. In Windows there is also
setlocale( LC_ALL, ".OCP" ); //Sets the locale to the current OEM code
page obtained from the operating system.
On Unix systems locales to some extend define the encoding being used. Windows, however, always uses UTF-16, and using locales for encoding issues does not make sense.
If fopen is to be
used the string should be transformed to the operating system locale.
This is obvious if you are using Windows. In Linux may not be that
obvious since all implementations always use UTF-8, or so I think.
The opposite is true: Windows never uses locale information for filenames (it always uses UTF-16 de facto), but the locale is used on Linux.
Just like mentioned above in the thread, the correct solution is to never use fopen (or other filesystem function from the C library), but always wrapper functions like g_fopen.
As an experiment, I replaced all fopen calls in LuaTeX with g_fopen, but I can't figure out how to link against GLib yet :-(