Perl newbie first experience with Unicode (in filename, -e operator, open operator, and cmd window)

Following code was tested in Windows 10 1903, perl -MWin32 -e"CORE::say Win32::GetACP()" returns ACP 1252 (Win 10 North America) with Win32 strawberry-perl 5.30.2.1 #1 Tue Mar 17 03:21:32 2020 x64.

Initial attempt to install cpan Win32::Unicode::File failed with t/04_print.t (Wstat: 768 Tests: 13 Failed: 3) message.

A quick search in Google lead to following post on Perl Monks. It looks like the problem with Win32::Unicode::File installation is known for some time.

NOTE: ikegami pointed out that the module can be forcefully installed and failed test can be ignored. Please see his comment bellow.

Following test code confirms that a forced installation cpan -f -i Win32::Unicode::File produces desired outcome.

use strict;
use warnings;
use feature 'say';

use utf8;

use Win32::Console;
use Win32::Unicode::File;

Win32::Console::OutputCP( 65001 );

binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';

my $fname = 'Доброе утро Россия.mkv';
my $fh = Win32::Unicode::File->new;

open $fh, '<:encoding(UTF-8)', $fname 
    or die "Can't open $fname $!";

while( <$fh> ) {
    say;
}

close $fh;

Content of input file Доброе утро Россия.mkv is

Доброе утро Россия

As suggested in above mentioned post I resorted to try Win32::LongPath as an alternative. Installation of the module went successfully through.

use strict;
use warnings;
use feature 'say';

use utf8;

use Win32::Console;
use Win32::LongPath;

Win32::Console::OutputCP( 65001 );

binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';

my $fname = 'IC3PEAK альбом Сладкая.mkv';
my $fh;

openL \$fh, '<:encoding(UTF-8)', $fname
    or die "Can't open $fname ($^E)";

while( <$fh> ) {
    # process input
    say;
}

close $fh;

Instead of real file IC3PEAK альбом Сладкая.mkv a text file with same name was used in the test with following content

Привет Москва

Note: use openL \$fh, '<', $fname on real mkv file to read content of the file

Three fixes are needed.

Non-ASCII source without use utf8;

Your source contains non-ASCII characters.

$media = "IC3PEAK альбом Сладкая.mkv";

Perl expects source code to be encoded using ASCII, unless you use use utf8;. Encode your source using UTF-8 and use use utf8;.

use utf8;

# String of decoded text (aka string of Unicode Code Points).
# Length = 26
my $media = "IC3PEAK альбом Сладкая.mkv";

Assuming your file was encoded using UTF-8, what you had was equivalent to the following:

use utf8;
use Encode qw( encode );

# String of text encoded using UTF-8 (aka string of bytes).
# Length = 39
my $media = encode("UTF-8", "IC3PEAK альбом Сладкая.mkv");

Incorrect output encoding

Your code contains

use open ":std", ":encoding(UTF-8)";

This tells Perl the following:

• Decode bytes received from STDIN using UTF-8.
• Encode characters sent to STDOUT and STDERR using UTF-8.
• Do the same for file handles opened in the current lexical scope.

The problem is that your terminal isn't expecting UTF-8. It's expecting cp437 (before chcp 1250) or cp1250 (after chcp 1250).

Solution 1:

Adjust the encoding specified in the use open line. This shows how this can be done without hardcoding the encoding.

Of course, you'll only be able to print the Cyrillic characters if the terminal's OEM code page (as set using chcp) supports the characters. This brings us to a second solution.

Solution 2:

Adjust the terminal to provide/expect UTF-8. This can be done using the following:

chcp 65001

Limitation of builtin functions that accept file names

Windows provides two versions of each functions that accepts strings:

• The "UNICODE" version (suffixed with "W" for "wide") accepts/returns strings encoded using UTF-16le. This version supports all Unicode characters.

• The "ANSI" version (suffixed with "A") accepts/returns strings encoded using the Active Code Page (ACP). The "A" version only supports a small subset of the Unicode characters.

  You can obtain the ACP for your system using the following:

  perl -MWin32 -e"CORE::say Win32::GetACP()"
  

Unfortunately, Perl functions (named operators) use the "A" version of system calls and expect/return text encoded using the ACP. This severely limits which file names that can be passed to them.

For example, my system's ACP is 1252, so the "A" version of system calls would not support Cyrillic characters. This means there is nothing I can do to make open, -e, etc work with file names containing Cyrillic characters. ouch.

[Upd: I now recommend Win32::LongPath instead.] The Win32-Unicode distribution can help with this. For example, -e is just a call to stat, and Win32::Unicode::File provides statW, a version of stat that accepts file names as decoded text. Similarly, it provides a replacement for open.