Find duplicate lines based on column and print both lines and their numbers with awk

user_2192002 0
Viewed 738

I have a following file:

userID PWD_HASH
test 1234
admin 1234
user 6789
abcd 5555
efgh 6666
root 1234

Using AWK, I need to find both original lines and their duplicates with row numbers, so that get the output like:

NR $0
1 test 1234
2 admin 1234
6 root 1234

I have tried the following, but it does not print the correct row number with NR :

awk 'n=x[$2]{print NR" "n;print NR" "$0;} {x[$2]=$0;}' file.txt

Any help would be appreciated!

duplicates awk linux
4 Answers

Using GAWK, you can do this by below construct : -

awk '
{
    NR>1
    {
       a[$2][NR-1 " " $0];
    }
}
END {
    for (i in a)
       if(length(a[i]) > 1)
          for (j in a[i])
             print j;
}
' Input_File.txt

Create a 2-dimensional array.

In first dimension, store PWD_HASH and in second dimension, store line number(NR-1) concatenated with whole line($0).

To display only duplicate ones, you can use length(a[i] > 1) condition.

user_19922760

Could you please try following.

awk '
FNR==NR{
  a[$2]++
  b[$2,FNR]=FNR==1?FNR:(FNR-1) OFS $0
  next
}
a[$2]>1{
  print b[$2,FNR]
}
'  Input_file  Input_file

Output will be as follows.

1 test 1234
2 admin 1234
6 root 1234

Explanation: Following is the explanation for above code.

awk '                                        ##Starting awk program here.
FNR==NR{                                     ##Checking condition here FNR==NR which will be TRUE when first time Input_file is being read.
  a[$2]++                                    ##Creating an array named a whose index is $1 and incrementing its value to 1 each time it sees same index.
  b[$2,FNR]=FNR==1?FNR:(FNR-1) OFS $0        ##Creating array b whose index is $2,FNR and concatenating its value to its own.
  next                                       ##Using next for skipping all further statements from here.
}
a[$2]>1{                                     ##Checking condition where value of a[$2] is greater than 1, this will be executed when 2nd time Input_file read.
  print b[$2,FNR]                            ##Printing value of array b whose index is $2,FNR here.
}
'  Input_file  Input_file                    ##Mentioning Input_file(s) names here 2 times.
user_58665800 
$ awk '
($2 in a) {          # look for duplicates in $2
    if(a[$2]) {      # if found
        print a[$2]  # output the first, stored one
        a[$2]=""     # mark it outputed
    }
    print NR,$0      # print the duplicated one
    next             # skip the storing part that follows
}
{
    a[$2]=NR OFS $0  # store the first of each with NR and full record
}' file

Output (with the header in file):

2 test 1234
3 admin 1234
7 root 1234
user_41623560

Without using awk, but GNU coretutils tools:

tail -n+2 file | nl | sort -k3n | uniq -D -f2

tail remove the first line.
nl add line number.
sort based on the 3rd field.
uniq only prints duplicate based on the 3rd field.

user_22146950
Related