Which encoding works best for Windows API calls?

Viewed 111

I use a function from the Windows API called GetFileAttributesW to retrieve attributes from a file. The function signature is defined as:

DWORD GetFileAttributesW([in] LPCWSTR lpFileName);

LPCWSTR is defined as const wchar_t*.

I want to call the function:

fs::path inputPath = ...
GetFileAttributesW(inputPath.whichMethodHere()?);

The given input path is of type std::filesystem::path that has several convenience converters like:

std::filesystem::path::string()
std::filesystem::path::wstring()
std::filesystem::path::u8string()
std::filesystem::path::u16string()
std::filesystem::path::u32string()

The two functions wstring() and u16string() stand out here for me. Both are defined with a character type of wchar_t and char16_t respectively.

Question 1:

What is the main difference between wstring() and u16string(). Does wstring() return something different than u16string in real life scenarios?

Question 2:

Which encoding does the Windows API generally expect?

2 Answers

The difference between std::wstring and std::u16string is that std::wstring uses wchar_t and std::u16string uses char16_t, which you managed to figure out by yourself.

wchar_t is not required to use UTF-16 encoding. Linux uses UTF-32 for wchar_t for example. wchar_t is only required to be able to store 'wide' strings, whatever that means, while char16_t is required to be able to store UTF-16 encoded text everywhere.

As the parameter is LPCWSTR the expected type of that parameter is const wchar_t*, which you also managed to figure out by yourself.

Windows functions which ends in A expects char pointers, and windows functions which ends in W expects wchar_t pointers. There are also macros which does not end in either, and these macros expand to either the A version or the W version depending on whether unicode is enabled during compilation.

Note: There is also std::filesystem::path::native() if you want the string as the native string type.

My recommendation would be GetFileAttributesW(inputPath.c_str()); as c_str returns a 'native' pointer.

TL;DR: char16_t doesn't provide any substantial advantage over wchar_t on Windows, and it's less convenient to use. Choose wchar_t, always. Microsoft's implementation of path::native() returns a std::wstring (not a std::u16string) as a matter of a conscious decision.


Which encoding does the Windows API generally expect?

Windows uses UTF-16LE encoding internally, everywhere, but doesn't enforce well-formedness anywhere. Particularly when dealing with filesystem objects, any sequence of 16-bit values (with a few exceptions) is admissible.

What is the main difference between wstring() and u16string(). Does wstring() return something different than u16string in real life scenarios?

The answer is more involved than this proposed answer suggests. wstring() and u16string() return a std::wstring and std::u16string, respectively, which are std::basic_string instantiations for wchar_t and char16_t.

The question thus boils down to this: What is the difference between wchar_t and char16_t? The answer is disturbingly unspecific. The section on character types lists them as following:

  • wchar_t - type for wide character representation (see wide strings). Required to be large enough to represent any supported character code point. It has the same size, signedness, and alignment as one of the integer types, but is a distinct type.

  • char16_t - type for UTF-16 character representation, required to be large enough to represent any UTF-16 code unit (16 bits). It has the same size, signedness, and alignment as std::uint_least16_t, but is a distinct type.

If you read carefully, neither type has a fixed size, and neither type is required to use any particular character encoding. It is specifically true that C++ makes no guarantees that char16_t must store UTF-16 encoded text (only that the type must be able to).

What does this mean in practice? Since Windows uses UTF-16LE everywhere, wchar_t has historically always been a 16-bit type on Windows. The introduction of char16_t on Windows hasn't changed much: It's just another type capable of representing UTF-16 encoded text. Since it is a distinct type, you're going to have to cast when passing a char16_t* into a function (which has always accepted a wchar_t*, or, rather, one of its C approximations).

On Windows, you can pretty much assume that wchar_t and char16_t will be the same width, and hold UTF-16 encoded characters. You can thus also assume, that wstring() and u16string() will return the same binary data. wchar_t is generally more useful as that's the native type used in interfaces, and doesn't require casts.

Related