Fast alternative to grep -f

user_1421408 0
Viewed 3859

file.contain.query.txt

ENST001

ENST002

ENST003

file.to.search.in.txt

ENST001  90

ENST002  80

ENST004  50

Because ENST003 has no entry in 2nd file and ENST004 has no entry in 1st file the expected output is: 

ENST001 90

ENST002 80

To grep multi query in a particular file we usually do the following:

grep -f file.contain.query <file.to.search.in >output.file

since I have like 10000 query and almost 100000 raw in file.to.search.in it takes very long time to finish (like 5 hours). Is there a fast alternative to grep -f ?

perl awk
8 Answers

This may be a little dated, but is tailor-made for simple UNIX utilities. Given:

  • keys are fixed-length (here 7 chars)
  • files are sorted (true in the example) allowing the use of fast merge sort

Then:

$ sort -m file.contain.query.txt file.to.search.in.txt | tac | uniq -d -w7

ENST002  80

ENST001  90

Variants:

To strip the number printed after the key, remove tac command:

$ sort -m file.contain.query.txt file.to.search.in.txt | uniq -d -w7

To keep sorted order, add an extra tac command at the end:

$ sort -m file.contain.query.txt file.to.search.in.txt | tac | uniq -d -w7 | tac
user_112037950
Related