Merge and remove redundant lines *among* files

user_7025817 0
Viewed 128

I need to merge several files, removing redundant lines among files, while keeping redundant lines within files. A schematic representation of my files is the following:

File1.txt

1
2
3
3
4
5
6

File2.txt

6
7
8
8
9

File3.txt

9
10
10
11

The desired output would be:

1
2
3
3
4
5
6
7
8
8
9
10
10
11

I would prefer to get a solution either in awk, or in bash or in R language. I searched the web for solutions and, though there were plenty of them* (please find some examples below), there were all removing duplicated lines regardless of the fact that they were located within or outside files.

Thanks in advance. Arturo

r merge awk bash
2 Answers

With your shown samples, could you please try following. This will NOT remove redundant lines within files but will remove them file wise.

awk '
FNR==1{
  for(key in current){
    total[key]
  }
  delete current
}
!($0 in total)
{
  current[$0]
}
' file1.txt file2.txt  file3.txt

Explanation: Adding detailed explanation for above.

awk '                                ##Starting awk program from here.
FNR==1{                              ##Checking condition if its first line(of each file) then do following.
  for(key in current){               ##Traverse through current array here.
    total[key]                       ##placing index of current array into total(for all files) one.
  }
  delete current                     ##Deleting current array here.
}
!($0 in total)                       ##If current line is NOT present in total then do following.
{
  current[$0]                        ##Place current line into current array.
}
' file1.txt file2.txt  file3.txt     ##Mentioning Input_file names here.
user_58665800

Here's a trick adding on to https://stackoverflow.com/a/15385080/3358272 using diff and its output format. There is likely a presumption of "sorted" here, untested.

out=$(mktemp -p .)
tmpout=$(mktemp -p .)
trap 'rm -f "${out}" "${tmpout}"' EXIT
for F in ${@} ; do
    { cat "${out}" ;
      diff --changed-group-format='%>' --unchanged-group-format='' "${out}" "${F}" ;
    } > "${tmpout}"
    mv "${tmpout}" "${out}"
done
cat "${out}"

Output:

$ ./question.sh F*
1
2
3
3
4
5
6
7
8
8
9
10
10
11

$ diff <(./question.sh F*) Output.txt

(Per markp-fuso's comment, if File3.txt had two 9s, this would preserve both.)

user_33582720
Related