remove duplicateds words for multiple strings in bash

user_11402476 0
Viewed 89

I would like to know how can I remove duplicate words from every line in bash with sed, awk, etc...

I have this file with 2000 lines, and I would like to know how can I keep one unique word per line:

OG0000005 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373 K00373  K00373
OG0000006 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374 K00374  K00374
OG0000007 K03089  K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089 K03089
OG0000008 K15554  K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599 K15599
OG0000009 K15555  K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555 K15555
OG0000010 K00817  K09758 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817 K00817
OG0000011 K07267  K07267  K07267  K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267 K07267
OG0000012 K22397  K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714 K01714
OG0000013 K00370  K07812 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370 K00370

So, the output should be like:

OG0000005 K00373
OG0000006 K00374
OG0000007 K03089  
OG0000008 K15554  K15599 
OG0000009 K15555 
OG0000010 K00817  K09758

I tried with:

sort file | uniq

wile read line
do
sort && uniq
done < file
sed awk bash
4 Answers

You may use this awk solution:

awk '
{
   delete seen
   printf "%s", $1
   for (i=2; i<=NF; ++i)
      if (!seen[$i]++)
         printf "%s", OFS $i
   print ""
}' file

OG0000005 K00373
OG0000006 K00374
OG0000007 K03089
OG0000008 K15554 K15599
OG0000009 K15555
OG0000010 K00817 K09758
OG0000011 K07267
OG0000012 K22397 K01714
OG0000013 K00370 K07812
user_5482250

This might work for you (GNU sed):

sed -E ':a;s/(( +\S+)\>.*)\2\>/\1/;ta' file

Replace a string which begins with a word which is later repeated by the orginal string less the repeated word.

Repeat until failure.

user_9674920

A pure Bash solution might be:

while read -r line; do 
    read -r -a a <<< "${line}"
    declare -A b
    for i in "${a[@]:1}"; do b["$i"]=1; done
    printf '%s %s\n' "${a[0]}" "${!b[*]}"
    unset b
done <file

To make the Bash + sort + uniq approach work, you might do:

while read -r line; do 
    read -r -a a <<< "${line}"
    re=$(tr ' ' '\n' <<< "${a[@]:1}" | sort | uniq | tr '\n' ' ' | xargs)
    printf "%s %s\n" "${a[0]}" "${re}"
done <file  
# if supported by your sort, you can also do 
# re=$(tr ' ' '\n' <<< "${a[@]:1}" | sort -u | tr '\n' ' ' | xargs)

Either prints:

OG0000005 K00373 
OG0000006 K00374 
OG0000007 K03089 
OG0000008 K15554 K15599 
OG0000009 K15555 
OG0000010 K00817 K09758 
OG0000011 K07267 
OG0000012 K22397 K01714 
OG0000013 K00370 K07812
user_2986070

Another solution without sed or awk, if you don't care about the original order of the words, might be:

cat file | xargs -I _ sh -c "echo _ | tr ' ' '\n' | sort | uniq | tr '\n' ' '; echo"

which outputs:

K00373 OG0000005 
K00374 OG0000006 
K03089 OG0000007 
K15554 K15599 OG0000008 
K15555 OG0000009 
K00817 K09758 OG0000010 
K07267 OG0000011 
K01714 K22397 OG0000012 
K00370 K07812 OG0000013

Else, if the first word has a special meaning and you want to keep it in its position, a solution might be to also use cut and paste in this way:

cat file | cut -d' ' -f1 --complement | xargs -I _ sh -c "echo _ | tr ' ' '\n' | sort | uniq | tr '\n' ' '; echo" | paste -d' ' <(cut -d' ' -f1 file) -

which outputs:

OG0000005 K00373 
OG0000006 K00374 
OG0000007 K03089 
OG0000008 K15554 K15599 
OG0000009 K15555 
OG0000010 K00817 K09758 
OG0000011 K07267 
OG0000012 K01714 K22397 
OG0000013 K00370 K07812
user_95477020
Related