AWK to extract first two lines with same word in a column

user_14361822 0
Viewed 66

I need to extract from a big multicolumn file (500mb to 1gb, \t separator) the first two lines containing a list of 100.000+ words appearing in a specific column.

Now I'm using a cycle like this one:

while read GREP
do
grep -m 2 "${GREP}" input.txt > output.txt ;
done < list_of_words.txt

but it takes too much time (i need to do this for many files), so i'm looking for alternatives.

a simple fgrep -f -m2 don't work since -m applies to the whole output of merged hits.

I suppose awk may be a solution but could not find help online.

For example if the input file is:

Dog Bird House
Mouse Giraffe Cat
Mouse Rhino House
Lion Horse House
Dog Rat Cat
Dog Mice Cat

I'd like to have an output file like this (Order of lines isn't relevant):

Dog Bird House
Mouse Rhino House
Mouse Giraffe Cat
Dog Rat Cat

I'm now using a list of word like this one:

House
Cat

But if the is a method to just keep the first two lines where the same word on column three would be even better! Note: words on column three will be unique and not appear in any of the other columns!

tabs ubuntu grep awk
1 Answers

Something like this?:

$ awk -F"\t" 'NR==FNR{a[$0]=2;next}($3 in a)&&--a[$3]>=0' list input

Output:

Dog     Bird    House
Mouse   Giraffe Cat
Mouse   Rhino   House
Dog     Rat     Cat

Explained:

$ awk -F"\t" '           # yes awk yes, fields tab delimited
NR==FNR {                # process the first file, list of words
    a[$0]=2              # hash every word, set initial value to 2
    next                 # on to the next word
}                        # process second file below this point
($3 in a) && --a[$3]>=0  # if 3rd field word in a and seen max once, output
' list input             # mind the file order
user_41623560
Related