What is the optimal multiplatform way of dealing with Unicode strings under C++?

I would strongly recommend using UTF-8 internally in your application, using regular old char* or std::string for data storage. For interfacing with APIs that use a different encoding (ASCII, UTF-16, etc.), I'd recommend using libiconv, which is licensed under the LGPL.

Example usage:

class TempWstring
{
public:
  TempWstring(const char *str)
  {
    assert(sUTF8toUTF16 != (iconv_t)-1);
    size_t inBytesLeft = strlen(str);
    size_t outBytesLeft = 2 * (inBytesLeft + 1);  // worst case
    mStr = new char[outBytesLeft];
    char *outBuf = mStr;
    int result = iconv(sUTF8toUTF16, &str, &inBytesLeft, &outBuf, &outBytesLeft);
    assert(result == 0 && inBytesLeft == 0);
  }

  ~TempWstring()
  {
    delete [] mStr;
  }

  const wchar_t *Str() const { return (wchar_t *)mStr; }

  static void Init()
  {
    sUTF8toUTF16 = iconv_open("UTF-16LE", "UTF-8");
    assert(sUTF8toUTF16 != (iconv_t)-1);
  }

  static void Shutdown()
  {
    int err = iconv_close(sUTF8toUTF16);
    assert(err == 0);
  }

private:
  char *mStr;

  static iconv_t sUTF8toUTF16;
};

iconv_t TempWstring::sUTF8toUTF16 = (iconv_t)-1;

// At program startup:
TempWstring::Init();

// At program termination:
TempWstring::Shutdown();

// Now, to convert a UTF-8 string to a UTF-16 string, just do this:
TempWstring x("Entr\xc3\xa9""e");  // "Entrée"
const wchar_t *ws = x.Str();  // valid until x goes out of scope

// A less contrived example:
HWND hwnd = CreateWindowW(L"class name",
                          TempWstring("UTF-8 window title").Str(),
                          dwStyle, x, y, width, height, parent, menu, hInstance, lpParam);

Same as Adam Rosenfield answer (+1), but I use UTFCPP instead.

I was recently on a project that decided to use std::wstring for a cross-platform project because "wide strings are Unicode, right?" This led to a number of headaches:

• How big is the scalar value in a wstring? Answer: It's up to the compiler implementation. In Visual Studio (Win), it is 16 bits. But in Xcode (Mac), it is 32 bits.
• This led to an unfortunate decision to use UTF-16 for communication over the wire. But which UTF-16? There are two: UTF-16BE (big-endian) and UTF16-LE (little-endian). Not being clear on this led to even more bugs.

When you are in platform-specific code, it makes sense to use the platform's native representation to communicate with its APIs. But for any code that is shared across platforms, or communicates between platforms, avoid all ambiguity and use UTF-8.

Rule of thumb: use the native platform Unicode form for processing (UTF-16 or UTF-32), and UTF-8 for data interchange (communication, storage).

If all the native APIs use UTF-16 (for instance in Windows), having your strings as UTF-8 means you will have to convert all input to UTF-16, call the Win API, then convert the answer to UTF-8. Quite a pain.

But if the main problem is the UI, the strings are the simple problem. The more difficult one is the UI framework. And for that I would recommend wxWidgets (http://www.wxWidgets.org). Supports many platforms, mature (17 years and still very active), native widgets, Unicode, liberal license.

I'd go for UTF16 representation in memory and UTF-8 or 16 on harddisk or wire. The main reason: UTF16 has a fixed size for each "letter". This simplifies a lot of duties when working with the string (searching, replacing parts, ...).

The only reason for UTF-8 is the reduced memory usage for "western/latin" letters. You can use this representation for disc-storage or transportation over network. It has also the benefit that you need not worry over byte-order when loading/saving to disc/wire.

With these reasons in mind, I'd go for std::wstring internally or - if your GUI library offers a Widestring, use that (like QString from QT). And for disc-storage, I'd write a small platform independent wrapper for the platform api. Or I'd check out unicode.org if they have platformindependent code available for this conversion.

for clarification: korean / japanese letters are NOT western / latin. Japanese are for exampli Kanji. That's why I mentioned the latin character set.

for UTF-16 not being 1 character / 2 bytes. This assumption is only true for characters being on the base multilingual plane (see: http://en.wikipedia.org/wiki/UTF16). Still most user of UTF-16 assume that all characters are on the BMP. If this can't be guaranteed for your application, you can switch to UTF32 or switch to UTF8.

Still UTF-16 is used for the reasons mentioned above in a lot of APIs (e.g. Windows, QT, Java, .NET, wxWidgets)

You can store UTF-16 inside std::string. So in principle you could use std::string for all platforms, and store inside the encoding preferred by the platform (UTF-8 for Linux, UTF-16 for Windows, etc.). This will leave you with something simple at the C++ types level, but having to track the encoding of strings. This may be simple if the application is self-contained, and less simple if it has to interoperate (cf. storage, wire format).

The risk of storing UTF-16 inside std::string is that sooner or later you will call .c_str() and the result will be interpreted as ending at the first 0, which for std::string s = reinterpret_cast<char *>(L"hello") will be at s[1].