regex to match special character

user_1118630 0
Viewed 150

There're many special characters in one text file(Line Terminator: LF; File encoding: utf-8) I'm processing, two of which are and . Their corresponding hex codes are \xf4\x80\x91\x9a and \xf4\x80\x91\x9d.

For testing purpose, you can put the following text into a text file 1.txt: a and a at the line end or you can use this file: https://drive.google.com/file/d/1E-8oZaLb86x0JE_gFpTkeX9jrbh3OXbF/view?usp=sharing

In editors like Sublime, I can't match these special characters using their hex codes. Not sure if there're other ways to do that.

With perl, I cannot match them either. I want to delete all these hamburger-like characters using regex:

perl -Mutf8::all -pE's,\xf4\x80\x91\x9a,,g; s,\xf4\x80\x91\x9d,,g;' 1.txt > 2.txt

Is there some way I can do that?

perl regex
1 Answers

Can you try to read the file as bytes/binary (using :raw IO layer):

use feature qw(say);
use strict;
use warnings;

my $fn = 'test.txt';
open ( my $fh, '<:raw', $fn ) or die "Could not open file '$fn': $!";
my $txt = do { local $/; <$fh> };
close $fh;
my @replace = ("\xf4\x80\x91\x9a", "\xf4\x80\x91\x9d");
my ($pat ) = map {qr/$_/} join "|", map quotemeta, @replace;
$txt =~ s/$pat//g;
print $txt;
user_21737730
Related