I've been frustrated for long by Windows code page issues, and the C programs portability and localisation issues they cause. The previous posts have detailed the issues at length, so I'm not going to add anything in this respect.
To make a long story short, eventually I ended up writing my own UTF-8 compatibility library layer over the Visual C++ standard C library. Basically this library ensures that a standard C program works right, in any code page, using UTF-8 internally.
This library, called MsvcLibX, is available as open source at https://github.com/JFLarvoire/SysToolsLib. Main features:
- C sources encoded in UTF-8, using normal char[] C strings, and standard C library APIs.
- In any code page, everything is processed internally as UTF-8 in your code, including the main() routine argv[], with standard input and output automatically converted to the right code page.
- All stdio.h file functions support UTF-8 pathnames > 260 characters, up to 64 KBytes actually.
- The same sources can compile and link successfully in Windows using Visual C++ and MsvcLibX and Visual C++ C library, and in Linux using gcc and Linux standard C library, with no need for #ifdef ... #endif blocks.
- Adds include files common in Linux, but missing in Visual C++. Ex: unistd.h
- Adds missing functions, like those for directory I/O, symbolic link management, etc, all with UTF-8 support of course :-).
More details in the MsvcLibX README on GitHub, including how to build the library and use it in your own programs.
The release section in the above GitHub repository provides several programs using this MsvcLibX library, that will show its capabilities. Ex: Try my which.exe tool with directories with non-ASCII names in the PATH, searching for programs with non-ASCII names, and changing code pages.
Another useful tool there is the conv.exe program. This program can easily convert a data stream from any code page to any other. Its default is input in the Windows code page, and output in the current console code page. This allows to correctly view data generated by Windows GUI apps (ex: Notepad) in a command console, with a simple command like: type WINFILE.txt | conv
This MsvcLibX library is by no means complete, and contributions for improving it are welcome!