Powershell find non-ASCII characters in text file

user_2653316 0
Viewed 7609

I am trying to find a way using Powershell Script to do the following.

  1. For each line in text file, check if line contains non-ASCII characters
  2. If line contains non-ASCII characters, output to separate file
  3. If line does not contain non-ASCII characters, skip to next line

By non-ASCII characters, I'm referring to non keyboard characters, e.g. accented characters, characters from another language, etc.

Sample Data

 - 张伟
 - குழந்தைகளுக்கான பெயர்கள்
 - 日本人の氏名
 - Full Name
 - Léna Rémi

Output Data

 - 张伟
 - குழந்தைகளுக்கான பெயர்கள்
 - 日本人の氏名
 - Léna Rémi

I found the regex in other threads to remove non-ASCII characters but I couldn't seem to make it work.

Please help!

** EDIT ** Thanks everyone for the help! I have managed to do what I wanted with the below script.

$nonASCII = "[^\x00-\x7F]"
foreach ($_ in [System.IO.File]::ReadLines($source)){
    if ($_ -cmatch $nonASCII){
        write-output $_ | out-File $output -append        
    }
}
non-ascii-characters powershell
3 Answers

Define a character set that describes all ASCII characters (code points 32 through 127 == [\x20-\x7F]), then negate it with ^ to match any non-ASCII character!

Let's test it against my (non-ASCII) name:

PS C:\> 'Mathias R. Jessen' -cmatch '[^\x20-\x7F]'
False
PS C:\> 'Mathias Rørbo Jessen' -cmatch '[^\x20-\x7F]'
True

To filter a list of strings, simply use the -cmatch operator in filter mode:

$strings = 'குழந்தைகளுக்கான பெயர்கள்', 'Boring John Doe', 'Léna Rémi'

$nonASCIIstrings = @($strings) -cmatch '[^\x20-\x7F]'

Or if you want to filter along a pipeline, use Where-Object:

$strings |Where-Object {$_ -cmatch '[^\x20-\x7F]'}
user_7126490

Here's a script I have to remove non-ascii characters from an xml file. Maybe you can use it as a starting point. I'm removing characters that are not between space and tilde in the ascii table, and also not tab. To me, ascii is in the range 0-127. Get-content takes out the carriage returns and linefeeds.

(get-content $args[0]) -replace '[^ -~\t]' | set-content $args[0]
user_66549420

The .NET regex engine supports a direct expression of the concept "non-ASCII character": \P{IsBasicLatin} (the inverse, i.e. "ASCII character", is \p{IsBasicLatin}):

' - 张伟',
' - குழந்தைகளுக்கான பெயர்கள்',
' - 日本人の氏名',
' - Full Name', 
' - Léna Rémi' -match '\P{IsBasicLatin}'

IsBasicLatin is an example of a named (Unicode) block.

The above uses -match, the regular-expression matching operator, to output those input lines (array elements) that contain at least one non-ASCII-range character:

 - 张伟
 - குழந்தைகளுக்கான பெயர்கள்
 - 日本人の氏名
 - Léna Rémi

For a streaming solution - processing lines read from a file one by one, you can combine -match with the Where-Object cmdlet:

Get-Content in.txt | 
  Where-Object { $_ -match '\P{IsBasicLatin}' } |
    Set-Content -Encoding Utf8 out.txt

Note that Get-Content is used to read the file line by line - while System.IO.File]::ReadLines("$pwd\in.txt") works too, it is only necessary if there's a performance problem.

user_453750
Related