Support UTF-8 locale for `std::filesystem::path`

[huangqinjin]

#include <filesystem>
#include <iostream>
#include <clocale>
#include <windows.h>

int main(int argc, char* argv[])
{
	SetConsoleOutputCP(CP_UTF8);
	std::setlocale(LC_CTYPE, ".65001");
	std::cout << std::filesystem::path(L"中文").string() << std::endl;
}

The code above compiled with llvm-mingw results in error at runtime:

libc++abi: terminating due to uncaught exception of type std::__1::__fs::filesystem::filesystem_error: filesystem error: in __wide_to_char: Illegal byte sequence

With clang-cl, it correctly prints the path to Console. Checked MS STL source code, it retrieves UTF-8 code page from ___lc_codepage_func.
https://github.com/microsoft/STL/blob/cbe2ee99caf122555a305d18f69e57c75b3fe5ec/stl/src/filesystem.cpp#L232-L244

[alvinhochun]

On Windows there are many encoding options, including: C locale codepage, console codepages (SetConsoleCP, SetConsoleOutputCP), ANSI codepage (ACP), and OEM codepage (OEMCP).

libc++ seems to use either ACP or OEMCP depending on the value of AreFileApisANSI(): https://github.com/llvm/llvm-project/blob/4a35c75e6590eacff33160fd7730e268090ae7d2/libcxx/src/filesystem/path.cpp#L427

For Windows, char paths are typically in ACP (or OEMCP if you called SetFileApisToOEM) considering that you often want to use it with Windows APIs. With this in mind, IMO it doesn't feel quite right for std::filesystem::path::string to use the locale codepage.

On the other hand, the MS docs on setlocale does say that "UTF-8 mode is also enabled for functions that have historically translated char strings using the default Windows ANSI code page (ACP)", and uses _mkdir and _getcwd as examples. Since one may also want to use std::filesystem::path::string with C APIs (that are not Windows APIs), this seems to be a valid reason for it to use the locale codepage if (and only if) it is UTF-8. However, this will be a breaking change for libc++, and there are probably other libc++ features that needs changing.

---

Just going by cppreference.com's page on std::filesystem::path::string, it states (emphasis mine):

Otherwise, if path::value_type is wchar_t, conversion, if any, is unspecified. This is the case on Windows, where wchar_t is 16 bit and the native encoding is UTF-16.

So there is no guarantee whether it should convert by the C locale codepage or ACP/OEMCP. (For practical purposes the console CP should be irrelevant.)

---

If you really want UTF-8 everywhere in your program, I'd recommend adding a manifest to the EXE to set ACP to UTF-8 with the activeCodePage field, which is supported on Windows 10 1903 or newer: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page

Otherwise, maybe you can use std::filesystem::path::u8string as a workaround for now.