grepping binary files and UTF16

taw_52751 0
Viewed 60181

Standard grep/pcregrep etc. can conveniently be used with binary files for ASCII or UTF8 data - is there a simple way to make them try UTF16 too (preferably simultaneously, but instead will do)?

Data I'm trying to get is all ASCII anyway (references in libraries etc.), it just doesn't get found as sometimes there's 00 between any two characters, and sometimes there isn't.

I don't see any way to get it done semantically, but these 00s should do the trick, except I cannot easily use them on command line.

utf-16 grep unicode
10 Answers

ripgrep

Use ripgrep utility to grep UTF-16 files.

ripgrep supports searching files in text encodings other than UTF-8, such as UTF-16, latin-1, GBK, EUC-JP, Shift_JIS and more. (Some support for automatically detecting UTF-16 is provided. Other text encodings must be specifically specified with the -E/--encoding flag.)

Example syntax:

rg sometext file

To dump all lines, run: rg -N . file.

user_550750

ugrep (Universal grep) fully supports Unicode, UTF-8/16/32 input files, detects invalid Unicode to ensure proper results, displays text and binary files, and is fast and free:

ugrep searches UTF-8/16/32 input and other formats. Option --encoding permits many other file formats to be searched, such as ISO-8859-1 to 16, EBCDIC, code pages 437, 850, 858, 1250 to 1258, MacRoman, and KOI8.

See ugrep on GitHub for details.

user_9188600

You can use the following Ruby's one-liner:

ruby -e "puts File.open('file.txt', mode:'rb:BOM|UTF-16LE').readlines.grep(Regexp.new 'PATTERN'.encode(Encoding::UTF_16LE))"

For simplicity, this can be defined as the shell function like:

grep-utf16() { ruby -e "puts File.open('$2', mode:'rb:BOM|UTF-16LE').readlines.grep(Regexp.new '$1'.encode(Encoding::UTF_16LE))"; }

Then it be used in similar way like grep:

grep-utf16 PATTERN file.txt

Source: How to use Ruby's readlines.grep for UTF-16 files?

user_550750
Related