Remove all duplicated lines based on first column

user_11504468 0
Viewed 51

I have a file :

AX-172321889 4
AX-172321889 4
AX-172322243 2
AX-172322331 2
AX-172322347 2
AX-172322347 2
AX-172322347 2
AX-172322354 2
AX-172322383 2
AX-172322440 2
AX-172322719 7

I need to remove every duplicated or more lines. And not keeping one occurence for each of them. I know :

awk '!seen[$1]++' or sort -t'-' -k1,2n | uniq is pretty close of what I want, but this keeps one occurence.

I should get :

AX-172322243 2
AX-172322331 2
AX-172322354 2
AX-172322383 2
AX-172322440 2
AX-172322719 7
sed awk
3 Answers

This might work for you (GNU uniq):

uniq -u file

Or if it just the first field use:

uniq -uw 12 file

Belt and braces:

sort file | uniq -uw 12

A GNU sed solution:

sed -E 'H;$!d;x;s/(\n\S+ )\S+(\1\S+)+//g;s/.//' file
user_9674920

A working awk solution:

awk '{++freq[$1]; rec[$1] = $0}
END {for (f in freq) if (freq[f] == 1) print rec[f]}' file

AX-172322440 2
AX-172322354 2
AX-172322383 2
AX-172322243 2
AX-172322719 7
AX-172322331 2

If you want sorted values in output then you may use this gnu-awk solution:

awk '{++freq[$1]; rec[$1] = $0} 
END {PROCINFO["sorted_in"] = "@ind_num_asc";
for (f in freq) if (freq[f] == 1) print rec[f]}' file

AX-172322243 2
AX-172322331 2
AX-172322354 2
AX-172322383 2
AX-172322440 2
AX-172322719 7
user_5482250

Assuming your key values are grouped together like in the example you provided (if not, sort the input first):

$ awk '
    $1 != p1 { if (cnt==1) print p0; cnt=0 }
    { ++cnt; p1=$1; p0=$0 }
    END { if (cnt==1) print p0 }
' file
AX-172322243 2
AX-172322331 2
AX-172322354 2
AX-172322383 2
AX-172322440 2
AX-172322719 7
user_17450010
Related