awk command to filter three times or more data filter

user_10510908 0
Viewed 69

i have a data set tab seperated like this:

A  B  C  D
1  aaa 1 2
1  aaa 3 4
1  aaa 5 6
1  bbb 7 8
1  ccc 9 1
1  ccc 2 3
1  ddd 4 5
1  ddd 6 7
1  ddd 8 9
1  ddd 1 2

Desired output:

A  B  C  D
1  aaa 1 2
1  aaa 3 4
1  aaa 5 6
1  ddd 4 5
1  ddd 6 7
1  ddd 8 9
1  ddd 1 2

I have tried this:

awk '++a[$2]>3' test.tsv test.tsv > test-2.tsv

Unwanted output:

1   ddd 1   2
1   aaa 1   2
1   aaa 3   4
1   aaa 5   6
1   ccc 2   3
1   ddd 4   5
1   ddd 6   7
1   ddd 8   9
1   ddd 1   2
awk
2 Answers

You may try this 2 pass awk:

awk -F '\t' 'FNR==NR {freq[$2]++; next} freq[$2] >= 3' test.tsv{,}

1  aaa 1 2
1  aaa 3 4
1  aaa 5 6
1  ddd 4 5
1  ddd 6 7
1  ddd 8 9
1  ddd 1 2
user_5482250

With your shown samples(with single pass of Input_file), could you please try following, written and tested with GNU awk.

awk '
BEGIN{ FS=OFS="\t" }
FNR==1{
  print
  next
}
{
  count[$2]++
  line[$2]=(line[$2]?line[$2] ORS:"")$0
}
END{
  for(i in count){
    if(count[i]>=3){
       print line[i]
    }
  }
}' Input_file

Explanation: Adding detailed explanation for above.

awk '                   ##Starting of awk program from here.
BEGIN{ FS=OFS="\t" }    ##Starting BEGIN section of this program from here.
                        ##Setting FS and OFS as tab here.
FNR==1{                 ##Checking condition if this is first line then do following.
  print                 ##Printing the current line here.
  next                  ##next will skip all further statements from here.
}
{
  count[$2]++           ##Creating count with index of 2nd field and keep increasing its count here.
  line[$2]=(line[$2]?line[$2] ORS:"")$0
                        ##Creating line array with index of 2nd field and keep adding lines to it with a new line.
}
END{                    ##Starting END block of this program from here.
  for(i in count){      ##Traversing through count array here.
    if(count[i]>=3){    ##Checking condition if count with index of i value is greater than or equals to 3 then do following.
       print line[i]    ##Printing value of line.
    }
  }
}' Input_file           ##Mentioning Input_file name here.
user_58665800
Related