Is there way to extract all the duplicate records based on a particular column?

Viewed 874

I'm trying to extract all (only) the duplicate values from a pipe delimited file.

My data file has 800 thousands rows with multiple columns and I'm particularly interested about column 3. So I need to get the duplicate values of column 3 and extract all the duplicate rows from that file.

I'm, however able to achieve this as shown below..

cat Report.txt | awk -F'|' '{print $3}' | sort | uniq -d >dup.txt

and I take the above in loop as shown below..

while read dup
do
   grep "$dup" Report.txt >>only_dup.txt
done <dup.txt

I've also tried the awk method

while read dup
do
awk -v a=$dup '$3 == a { print $0 }' Report.txt>>only_dup.txt
done <dup.txt

But, as I have large number of records in the file, it's taking ages to complete. So I'm looking for an easy and quick alternative.

For example, I have data like this:

1|learning|Unix|Business|Requirements
2|learning|Unix|Business|Team
3|learning|Linux|Business|Requirements
4|learning|Unix|Business|Team
5|learning|Linux|Business|Requirements
6|learning|Unix|Business|Team
7|learning|Windows|Business|Requirements
8|learning|Mac|Business|Requirements

And my expected output which doesn't include unique records:

1|learning|Unix|Business|Requirements
2|learning|Unix|Business|Team
4|learning|Unix|Business|Team
6|learning|Unix|Business|Team
3|learning|Linux|Business|Requirements
5|learning|Linux|Business|Requirements
4 Answers

This may be what you want:

$ awk -F'|' 'NR==FNR{cnt[$3]++; next} cnt[$3]>1' file file
1|learning|Unix|Business|Requirements
2|learning|Unix|Business|Team
3|learning|Linux|Business|Requirements
4|learning|Unix|Business|Team
5|learning|Linux|Business|Requirements
6|learning|Unix|Business|Team

or if the file's too large for all the keys ($3 values) to fit in memory (which shouldn't be a problem with just the unique $3 values from 800,000 lines):

$ cat tst.awk
BEGIN { FS="|" }
{ currKey = $3 }
currKey == prevKey {
    if ( !prevPrinted++ ) {
        print prevRec
    }
    print
    next
}
{
    prevKey = currKey
    prevRec = $0
    prevPrinted = 0
}

$ sort -t'|' -k3,3 file | awk -f tst.awk
3|learning|Linux|Business|Requirements
5|learning|Linux|Business|Requirements
1|learning|Unix|Business|Requirements
2|learning|Unix|Business|Team
4|learning|Unix|Business|Team
6|learning|Unix|Business|Team

EDIT2: As per Ed sir's suggestion fine tuned my suggestion with more meaningful names(IMO) of arrays.

awk '
match($0,/[^\|]*\|/){
  val=substr($0,RSTART+RLENGTH)
  if(!unique_check_count[val]++){
    numbered_indexed_array[++count]=val
  }
  actual_valued_array[val]=(actual_valued_array[val]?actual_valued_array[val] ORS:"")$0
  line_count_array[val]++
}
END{
  for(i=1;i<=count;i++){
    if(line_count_array[numbered_indexed_array[i]]>1){
      print actual_valued_array[numbered_indexed_array[i]]
    }
  }
}
'  Input_file

Edit by Ed Morton: FWIW here's how I'd have named the variables in the above code:

awk '
match($0,/[^\|]*\|/) {
  key = substr($0,RSTART+RLENGTH)
  if ( !numRecs[key]++ ) {
    keys[++numKeys] = key
  }
  key2recs[key] = (key in key2recs ? key2recs[key] ORS : "") $0
}
END {
  for ( keyNr=1; keyNr<=numKeys; keyNr++ ) {
    key = keys[keyNr]
    if ( numRecs[key]>1 ) {
      print key2recs[key]
    }
  }
}
' Input_file


EDIT: Since OP changed Input_file with |delimited so changing code a bit to as follows, which deals with new Input_file(Thanks to Ed Morton sir for pointing it out).

awk '
match($0,/[^\|]*\|/){
  val=substr($0,RSTART+RLENGTH)
  if(!a[val]++){
    b[++count]=val
  }
  c[val]=(c[val]?c[val] ORS:"")$0
  d[val]++
}
END{
  for(i=1;i<=count;i++){
    if(d[b[i]]>1){
      print c[b[i]]
    }
  }
}
'   Input_file


Could you please try following, following will give output in same sequence of in which lines are occurring in Input_file.

awk '
match($0,/[^ ]* /){
  val=substr($0,RSTART+RLENGTH)
  if(!a[val]++){
    b[++count]=val
  }
  c[val]=(c[val]?c[val] ORS:"")$0
  d[val]++
}
END{
  for(i=1;i<=count;i++){
    if(d[b[i]]>1){
      print c[b[i]]
    }
  }
}
'  Input_file

Output will be as follows.

2 learning Unix Business Team
4 learning Unix Business Team
6 learning Unix Business Team
3 learning Linux Business Requirements
5 learning Linux Business Requirements

Explanation for above code:

awk '                                 ##Starting awk program here.
match($0,/[^ ]* /){                   ##Using match function of awk which matches regex till first space is coming.
  val=substr($0,RSTART+RLENGTH)       ##Creating variable val whose value is sub-string is from starting point of RSTART+RLENGTH value to till end of line.
  if(!a[val]++){                      ##Checking condition if value of array a with index val is NULL then go further and increase its index too.
    b[++count]=val                    ##Creating array b whose index is increment value of variable count and value is val variable.
  }                                   ##Closing BLOCK for if condition of array a here.
  c[val]=(c[val]?c[val] ORS:"")$0     ##Creating array named c whose index is variable val and value is $0 along with keep concatenating its own value each time it comes here.
  d[val]++                            ##Creating array named d whose index is variable val and its value is keep increasing with 1 each time cursor comes here.
}                                     ##Closing BLOCK for match here.
END{                                  ##Starting END BLOCK section for this awk program here.
  for(i=1;i<=count;i++){              ##Starting for loop from i=1 to till value of count here.
    if(d[b[i]]>1){                    ##Checking if value of array d with index b[i] is greater than 1 then go inside block.
      print c[b[i]]                   ##Printing value of array c whose index is b[i].
    }
  }
}
'  Input_file                         ##Mentioning Input_file name here.

Another in awk:

$ awk -F\| '{                  # set delimiter
    n=$1                       # store number
    sub(/^[^|]*/,"",$0)        # remove number from string
    if($0 in a) {              # if $0 in a
        if(a[$0]==1)           # if $0 seen the second time
            print b[$0] $0     # print first instance
        print n $0             # also print current
    }
    a[$0]++                    # increase match count for $0
    b[$0]=n                    # number stored to b and only needed once
}' file

Output for the sample data:

2|learning|Unix|Business|Team
4|learning|Unix|Business|Team
3|learning|Linux|Business|Requirements
5|learning|Linux|Business|Requirements
6|learning|Unix|Business|Team

Also, would this work:

$ sort -k 2 file | uniq -D -f 1

or -k2,5 or smth. Nope, as the delimiter changed from space to pipe.

Two steps of improvement.
First step:
After

awk -F'|' '{print $3}' Report.txt | sort | uniq -d >dup.txt
# or
cut -d "|" -f3 < Report.txt | sort | uniq -d >dup.txt

you can use

grep -f <(sed 's/.*/^.*|.*|&|.*|/' dup.txt) Report.txt
# or without process substitution
sed 's/.*/^.*|.*|&|.*|/' dup.txt > dup.sed
grep -f dup.sed Report.txt

Second step:
Use awk as given in other, better, answers.

Related