How to print UTF-8 strings to std::cout on Windows?

user_52041 0
Viewed 45115

I'm writing a cross-platform application in C++. All strings are UTF-8-encoded internally. Consider the following simplified code:

#include <string>
#include <iostream>

int main() {
    std::string test = u8"Greek: αβγδ; German: Übergrößenträger";
    std::cout << test;

    return 0;
}

On Unix systems, std::cout expects 8-bit strings to be UTF-8-encoded, so this code works fine.

On Windows, however, std::cout expects 8-bit strings to be in Latin-1 or a similar non-Unicode format (depending on the codepage). This leads to the following output:

Greek: ╬▒╬▓╬│╬┤; German: ├£bergr├Â├ƒentr├ñger

What can I do to make std::cout interpret 8-bit strings as UTF-8 on Windows?

This is what I tried:

#include <string>
#include <iostream>
#include <io.h>
#include <fcntl.h>

int main() {
    _setmode(_fileno(stdout), _O_U8TEXT);
    std::string test = u8"Greek: αβγδ; German: Übergrößenträger";
    std::cout << test;

    return 0;
}

I was hoping that _setmode would do the trick. However, this results in the following assertion error in the line that calls operator<<:

Microsoft Visual C++ Runtime Library

Debug Assertion Failed!

Program: d:\visual studio 2015\Projects\utf8test\Debug\utf8test.exe File: minkernel\crts\ucrt\src\appcrt\stdio\fputc.cpp Line: 47

Expression: ( (_Stream.is_string_backed()) || (fn = _fileno(_Stream.public_stream()), ((_textmode_safe(fn) == __crt_lowio_text_mode::ansi) && !_tm_unicode_safe(fn))))

For information on how your program can cause an assertion failure, see the Visual C++ documentation on asserts.

utf-8 encoding windows c++
8 Answers

Forget everything you know about the Windows console and its Unicode/UTF-8 support (or rather lack of support). This is 2020 and it's a new world. This is not a direct answer to the question above, but rather an alternative that makes much more sense now, a new way that was not possible before.

Everybody's right, the root problem is the Windows console. But there's a new player in town, and it's Windows Terminal. Install and launch Windows Terminal. Use this program:

#include <iostream>
#include <windows.h>

int main()
{
    SetConsoleOutputCP(CP_UTF8); 
    // or have your user set the console codepage: `chcp 65001`
    
    std::cout << "\"u\" with two dots on top: \xc3\xbc\n";
    std::cout << "chinese glyph for \"world\": \xe5\x80\xbc\n";
    std::cout << "smiling emoji: \xf0\x9f\x98\x80\n";
    return 0;
}

This program sends UTF-8 through a plain cout.

The output:

Unicode output in Windows Terminal

The command chcp 65001 or SetConsoleOutputCP(CP_UTF8) is required for a cmd tab in Windows Terminal, but it looks like it is not in a Powershell tab. Maybe Powershell is UTF-8 by default?

Rooting out the core issue, cmd, is now the best option in my opinion. Spread the word.

user_54245860

Since I started using the {fmt} library, all my encoding problems are gone.

A simple example of use:

#include <fmt/core.h>

int main() {
  fmt::print("Greek: αβγδ; German: Übergrößenträger\n");
}
user_151388940

I had the same problem and wrote a very small library called libpu8 for this: https://github.com/jofeu/libpu8

For windows consoles, it replaces the streambufs of cin, cout and cerr so that they accept and produce utf-8 at the front end and talk to the console in UTF-16. On non-windows operating systems, or if cin, cout, cerr are attached to files/pipes and not consoles, it does nothing. It also translates the arguments of the C++ main() function to UTF-8 on windows.

Usage Example:

#include <libpu8.h>
#include <string>
#include <fstream>
#include <windows.h>

// argv are utf-8 strings when you use main_utf8 instead of main.
// main_utf8 is a macro. On Windows, it expands to a wmain that calls
// main_utf8 with converted strings.
int main_utf8(int argc, char** argv)
{
        // this will also work on a non-Windows OS that supports utf-8 natively
        std::ofstream f(u8widen(argv[1]));
        if (!f)
        {
                // On Windows, use the "W" functions of the windows-api together
                // with u8widen and u8narrow
                MessageBoxW(0,
                        u8widen(std::string("Failed to open file ") + argv[1]).c_str(), 0, 0);
                return 1;
        }
        std::string line;
        // line will be utf-8 encoded regardless of whether cin is attached to a
        // console, or a utf-8 file or pipe.
        std::getline(std::cin, line);
        // line will be displayed correctly on a console, and will be utf-8 if
        // cout is attached to a file or pipe.
        std::cout << "You said: " << line;
        return 0;
}
user_114588160
Related