UTF8 output from Strawberry Perl

user_6212907 0
Viewed 344

I have a text file test.txt with the UTF8 encoded content äöü (these are German umlauts and just an example, the file size is 6 Bytes). I also have a Cygwin terminal on a Windows 10 PC with the correct LANG settings:

$ cat test.txt
äöü

I'd like to print the content of this file with a Perl script, but can't get it to work.

open my $fh, '<', 'test.txt';
print <$fh>;
close $fh;

results in

$ perl test.pl
├ñ├Â├╝

I tried all variations I found at How can I output UTF-8 from Perl? - none of them solved my problem. What's wrong?

EDIT per request:

$ file test.txt
test.txt: UTF-8 Unicode text, with no line terminators

$ echo $LANG

I also tried setting LANG to de_DE.UTF-8.

EDIT to narrow down the problem: If I try this with the Perl version 5.32.1 included in Cygwin, it works as expected. It still doesn't work in Strawberry Perl version 5.32.1. So it's probably no Perl problem nor a Windows problem nor something with language or encoding settings, it's a Strawberry Perl problem.

cygwin utf-8 perl
4 Answers

If you are in a cmd.exe window or in PowerShell, you can change the codepage to 65001:

chcp 65001

If you do not want to change the codepage find out what chcp (or "cp".Win32::GetConsoleOutputCP()) returns and encode to that encoding.

use Encode;

open my $fh, '<:utf8','test.txt';
while(<$fh>){
    print encode('cp850',$_); # needs a recent Encode to support cp850
};
close $fh;

If you are in cygwin bash, you can call chcp with system() like so:

use strict;
use warnings;
use Encode;

system("chcp 65001 > NUL");

open my $fh, '<:utf8','test.txt';
while(<$fh>){   
    print encode('utf8',$_); # needs a recent Encode to support cp850
};
close $fh;
user_106678390

It seems you are missing the LANG setting

$ export LANG=de_DE.UTF-8

$ echo $LANG
de_DE.UTF-8

$ cat test.txt
äöü

$ perl test.pl
äöü

$ file test.txt
test.txt: UTF-8 Unicode text

$ od -c  test.txt
0000000 303 244 303 266 303 274  \n
0000007

$ which perl
/usr/bin/perl
user_51053830 
$ "$( cygpath 'C:\progs\sp5302-x64\perl\bin\perl.exe' )" -M5.010 -e'
   use Win32;
   BEGIN {
      Win32::SetConsoleCP(65001);
      Win32::SetConsoleOutputCP(65001);
   }
   use open ":std", ":encoding(UTF-8)";
   say chr(0x2660);
'
♠

(BEGIN { `chcp 65001` } would also have done the trick.)

user_5899240

You may explicitly define encoding of input and output.

open( my $fh, '<:utf8', 'test.txt');
binmode(STDOUT,':utf8');
print <$fh>;
close $fh;
user_21397660
Related