Remove all but one line if they have common text at the end

user_11776841 0
Viewed 41

this is very specific and couldn't find this case in the other answers.

This is the sample text:

exampleA/file_A.a
exampleB/file_A.a

exampleB/another_dir/file_B.a
exampleB/file_A.a
exampleA/file_C.a
exampleB/file_D.b

exampleB/file_C.a
exampleB/file_B.a
exampleA/another_dir/file_D.b
exampleA/another_dir/file_C.a
exampleB/another_dir/another_one/file_D.b

I want to delete the lines of the duplicated files with an specific extension (.a) that could appear in this list of files (text file) EXCEPT one, so the text contains only 1 line per file. But there can be more files than fileA.a, fileB.a and fileC.a, so I can't "hardcode" those.

How do I search the lines that contain the same file at the end? I managed to do this: ( the extension of the files I want to delete is always .a, and the names can contain underscores anywhere, and I dont want to delete the files with the extension .b, because they are intended to be the same and they're always in different folders )

grep -o '/[a-zA-Z0-9_]*\.[a]' file.txt | sort | uniq -d

But I don't have the line numbers.

And then, how do I delete those lines BUT one? I have seen in a question the next line:

awk '!seen[$0]++' file.txt

But I can't figure out how to combine these to have the output that I need, that should look like this:

exampleA/file_A.a

exampleB/another_dir/file_B.a
exampleA/file_C.a
exampleB/file_D.b
exampleA/another_dir/file_D.b
exampleB/another_dir/another_one/file_D.b

Thanks.

Edit: I forgot to mention that there are another files in the text that have another extension (let's say .b) and I don't want to touch those. I just want to delete the ones with an specific extension (.a) and maybe another one (.d) if they appear, but that is not ultra necessary. I edited the sample.

awk linux
2 Answers

You may use this awk:

awk -F/ 'NF && (!/\.a$/ || !seen[$NF]++)' file

exampleA/file_A.a
exampleB/another_dir/file_B.a
exampleA/file_C.a
exampleB/file_D.b
exampleA/another_dir/file_D.b
exampleB/another_dir/another_one/file_D.b

Here

  • -F/ sets / as input file separator
  • NF selects all non-empty lines
  • !/.a$/ || !seen[$NF]++: Prints a line if it doesn't end with .a` or if last field is read first time.
user_5482250 
awk -F/ '$NF!~/\.(a|d)$/|| !seen[$NF]++' file.txt
exampleA/file_A.a

exampleB/another_dir/file_B.a
exampleA/file_C.a
exampleB/file_D.b

exampleA/another_dir/file_D.b
exampleB/another_dir/another_one/file_D.b

should do what you want. Note that in your input sample you have file_C.a and fileC.a

What we do is to tell awk to use the / as a field separator and only use the filen-name portion, the last field $NF, as the array index.

user_13947290
Related