How to use "grep -f file" if "file" has null-delimited items?

user_6176817 0
Viewed 437

I need to find null-delimited items from numerous files (data2, data3, ...) that are present in data1. Exact match is required.

All works well with grep -f data1 data2 data3 ... until the items in data1 are also null-delimited.

  1. Using only newlines - ok:

    $ cat data1
1234
abcd
efgh
5678
$ cat data2
1111
oooo
abcd
5678
$ grep -xFf data1 data2
abcd
5678

  2. data2 contains null-delimited items - ok when -z used:

    $ printf '1111\0oooo\0abcd\0005678' > data2
$ grep -zxFf data1 data2 | xargs -0 printf '%s\n'
abcd
5678

  3. Now both data1 and data2 contain null-delimited items - fail. Seems that the -z option does not apply to the file specified with -f:

    $ printf '1234\0abcd\0efgh\0005678' > data1
$ grep -zxFf data1 data2 | xargs -0 printf '%s\n'

$

The problem is that I do need both files to have null-delimited items. Obvious work-around could be (for example) a good old while loop:

while IFS= read -rd '' line || [[ $line ]]; do
    if grep -zqxF "$line" data2; then
        printf '%s\n' "$line"
    fi
done < data1

But since I have many files with lots of items, this will be painfully slow! Is there a better approach (I do not insist on using grep)?

comm grep bash
2 Answers

Since order retention isn't important, you're trying to match exact strings, and you have GNU tools available, instead of using fgrep I'd suggest comm -z.

$ printf '%s\0' 1111 oooo abcd 005678 >data2
$ printf '%s\0' 1234 abcd efgh 005678 >data
$ comm -z12 <(sort -uz <data) <(sort -uz <data2) | xargs -0 printf '%s\n'
005678
abcd

If you generate your files sorted in the first place (and thus can leave out the sort operations), this will also have very good memory and performance characteristics.

user_141220

(Although the following may not be the best solution for this particular case, I added it anyway in case it helps some future reader with a similar problem. See below for a gawk solution which may be useful for this use case.)

grep has newline hard-wired as the pattern terminator. Even if you use -e pattern, newlines in the pattern string will cause grep to handle the options as specifying multiple patterns, not a single pattern containing newline characters.

However, if your NUL-separated patterns do not contain newline characters, you can use Gnu xargs and sed to construct an appropriate grep invocation with -e command-line arguments:

sed -z 's/^/-e/' data | xargs -0 grep -zF data2 ...

(This works because Gnu grep reshuffles command-line arguments, so it's ok to put the files to be searched before the patterns. That won't work on many other grep implementations.)

As far as I know, there is no workaround for patterns which may contain newline characters. grep -E and grep -F do not recgnise ascii escape sequences, and will silently create multiple patterns from a pattern which contains a newline. grep -P (another Gnu extension which uses PCRE regexen) will properly handle embedded newline characters or ascii escapes, but only allows a single pattern.

Full-line NUL-terminated matches without sorting

In the case that you are only interested in exact, full "line" matches (-Fx), you could use an Gnu Awk script rather than sorting the inputs and patterns. This could be a win for very large inputs which don't fit in memory; sorting with external temporary files can be quite expensive. The Awk solution uses a hash table, so sorting is unnecessary. (Again, this might not work on all Awks because it relies on setting RS to NUL.)

awk -v RS=`\0` 'NR==FNR{p[$0] = 1; next;} $0 in p' data data2 ...
user_15662210
Related