Shell command to collapse tabular data to make a non-redundant column

user_459780 0
Viewed 119

I have some tabular data that looks like this.

1f2edc31defc588a369544ac32215afec2ae65da  2019-01-08
e95c31b76be6b99eb8a0670b93b91c9e9abf2efb  2018-11-27
c2dd1d6be6434b6ef109fa0394546fa500501efe  2018-11-27
c6b6b534addff919150d94276f61eb8d0882f3f6  2018-09-04
459a495a929cffa2205d31c6cc7ba2750af2580b  2018-06-25
11f7883b29295e39bc964522deb96132d6a2803e  2018-06-25
73a2a72ee9d0a9acf0ed02b0959f3fb5a44c3fc7  2018-06-25
0970c0fef779a4ea89638140f339c2047d10d0c5  2017-12-29

I need to remove lines from the table so that no value is duplicated in the second (date) column--only the first occurrence should remain.

I haven't used Perl in a while, but after a couple google searches to remind myself of the correct syntax I was able to come up with this one-liner.

$ perl -e '%seen = (); for (<>) { @v = split(/\s+/); unless (exists($seen{$v[1]})) { $seen{$v[1]} = 1; print } }' < data.tsv > data.clean.tsv

After a quick glance this seems to do the trick, but it's a mess. Is there a cleaner more concise way to do this using portable (UNIX) shell commands, or is any solution going to involve a mess of Perl/sed/awk/etc.?

perl shell
3 Answers

Need not be messy

perl -ne'print unless $exists{(split)[1]}++' dup_data.txt
user_46533790

Perl anticipated problems like this and came up with concise solutions:

perl -ane 'print unless $seen{$F[1]}++' < data.csv > data.clean.csv

See perlrun for information about the -a and -n switches.

With the helpful B::Deparse module, you can see how Perl expands the command-line switches into something that resembles your original script.

$ perl -MO=Deparse -ane 'print unless $seen{$F[1]}++' 
LINE: while (defined($_ = <ARGV>)) {
    our(@F) = split(' ', $_, 0);
    print $_ unless $seen{$F[1]}++;
}
-e syntax OK
user_1686570

Since data.tsv is already sorted on the 2nd field, a simpler method, no perl needed:

uniq -f 1 data.tsv > data.clean.tsv

Output, (contents of data.clean.tsv):

1f2edc31defc588a369544ac32215afec2ae65da  2019-01-08
e95c31b76be6b99eb8a0670b93b91c9e9abf2efb  2018-11-27
c6b6b534addff919150d94276f61eb8d0882f3f6  2018-09-04
459a495a929cffa2205d31c6cc7ba2750af2580b  2018-06-25
0970c0fef779a4ea89638140f339c2047d10d0c5  2017-12-29
user_61362140
Related