Find duplicated rows (based on 2 columns) in Data Frame in R

Here's a dplyr option for tagging duplicates based on two (or more) columns. In this case ric and date:

df <- data_frame(ric = c('S1A.PA', 'ABC.PA', 'EFG.PA', 'S1A.PA', 'ABC.PA', 'EFG.PA'),
                 date = c('2011-06-30 20:00:00', '2011-07-03 20:00:00', '2011-07-04 20:00:00', '2011-07-05 20:00:00', '2011-07-03 20:00:00', '2011-07-04 20:00:00'),
                 open = c(23.7, 24.31, 24.495, 24.23, 24.31, 24.495))

df %>% 
  group_by(ric, date) %>% 
  mutate(dupe = n()>1)
# A tibble: 6 x 4
# Groups:   ric, date [4]
  ric    date                 open dupe 
  <chr>  <chr>               <dbl> <lgl>
1 S1A.PA 2011-06-30 20:00:00  23.7 FALSE
2 ABC.PA 2011-07-03 20:00:00  24.3 TRUE 
3 EFG.PA 2011-07-04 20:00:00  24.5 TRUE 
4 S1A.PA 2011-07-05 20:00:00  24.2 FALSE
5 ABC.PA 2011-07-03 20:00:00  24.3 TRUE 
6 EFG.PA 2011-07-04 20:00:00  24.5 TRUE 

Easy way to get the information you want is to use dplyr.

yourDF %>% 
  group_by(RIC, Date) %>% 
  mutate(num_dups = n(), 
         dup_id = row_number()) %>% 
  ungroup() %>% 
  mutate(is_duplicated = dup_id > 1)

Using this:

• num_dups tells you how many times that particular combo is duplicated
• dup_id tells you which duplicate number that particular row is (e.g. 1st, 2nd, or 3rd, etc)
• is_duplicated gives you an easy condition you can filter on later to remove all the duplicate rows (e.g. filter(!is_duplicated)), though you could also use dup_id for this (e.g. filter(dup_id == 1))

If you want to remove duplicate records based on values of Columns Date and State in dataset data.frame:

#Indexes of the duplicate rows that will be removed: 
duplicate_indexes <- which(duplicated(dataset[c('Date', 'State')]),) 
duplicate_indexes 

#new_uniq will contain unique dataset without the duplicates. 
new_uniq <- dataset[!duplicated(dataset[c('Date', 'State')]),] 
View(new_uniq) 

Found quite a masterful idea posted by Steve Lianouglou that helps solve this problem with the great advantage of indexing the repetitions:

If you generate a hash column concatenating both your columns for which you want to check duplicates, you can then use dplyr::n() together with seq to give an index to each duplicate occurrence as follows

dat %>% mutate(hash = str_c(RIC, Date)) %>%
  group_by(hash) %>% 
  mutate(duplication_id = seq(n()) %>%
 ungroup ()

Your column duplication_id tells you how many identical rows (same row values for both columns) are there in your table above the one indexed. I used this to remove duplicate Ids.