R: How to remove duplicates with some conditions in a complex Dataframe?

user_8826712 0
Viewed 82

I have a dataset for factory producing Gold and Silver product (Pen), we would like to check the quality by assigning employees to check these products produced from all machines in the factory. Data sample data below:

Every machine is in a specific Room/Section/Building, and we have two columns to group employee IDs that are testing Gold and Silver Pens.

The issue is I have duplicates employees testing the quality of the same machine. So I would like to remove these duplicates and group the ones which are not duplicates. Sample:

Bld.No <- c(1,1,1,1,1,1,2,2,2,2)
Section <- c("A","A","A","A","B","B","C","C","D","D")
Room.No <- c(100,100,100,100,200,200,300,300,400,400)
Gold <- c(8,6,4,0,6,0,7,2,2,1)
Silver <- c(1,0,0,1,2,3,4,0,4,0)
Total <- c(9,6,4,1,8,3,11,2,6,1)
Emp.Gold.ID <- c("A11, A09, B22, E12, A04, C09, D33, A01", "A11, A09, B22, E12, A04, A01", "A09, 822, E12, A04", NA, "A71, A09, B12, E32, A04, C19", NA, "B22, E12, A04, C09, D33, A01, M11", "E12, Z09", "C09, D33", "D18")
Emp.Silver.ID <- c("A17", NA, NA, "D33", "B22, E12", "A09, B12, E32", "A44, C02, D03, A71", NA, "A12, A01, M11, D18", NA)

df <- data.frame(Bld.No, Section, Room.No, Gold, Silver, Total, Emp.Gold.ID, Emp.Silver.ID)

Note: if emp.Id is already in the previous records, either gold or silver, we should remove it. Meaning ID should be in either one and remove the duplicate. See the example of the last record in the sample and output table, we removed the last record (2, D, 400, 1, 0, 1, D18, NA), because of D18 is already in the previous record, even though it's in the Silver column.

Sample Data and Output:

Sample Data and Output

dataframe duplicates r
1 Answers

To do this, I would use separate_rows to get all the IDs in separate rows to remove duplicates later on with distinct.

After removing duplicate IDs, would create comma separated strings of the IDs for gold and silver. You can either summarize total Gold and Silver before this step or afterwards.

Note that to get the same results as in your sample Data and Output, I changed 822 to B22.

Please let me know if this is what you had in mind.

library(dplyr)
library(tidyr)

df$Emp.Gold.ID <- as.character(df$Emp.Gold.ID)
df$Emp.Silver.ID <- as.character(df$Emp.Silver.ID)

df %>% 
  separate_rows(Emp.Gold.ID) %>%
  separate_rows(Emp.Silver.ID) %>%
  pivot_longer(cols = starts_with("Emp."), names_to = "ID", values_drop_na = TRUE) %>%
  group_by(Bld.No, Section, Room.No) %>%
  distinct(value, .keep_all = TRUE) %>%
  group_by(Bld.No, Section, Room.No, ID) %>%
  summarise(NewID = toString(value)) %>%
  pivot_wider(names_from = ID, values_from = NewID) %>%
  mutate(Gold = length(unlist(strsplit(Emp.Gold.ID, ", "))),
         Silver = length(unlist(strsplit(Emp.Silver.ID, ", "))),
         Total = Gold + Silver)

# A tibble: 4 x 8
# Groups:   Bld.No, Section, Room.No [8]
  Bld.No Section Room.No Emp.Gold.ID                            Emp.Silver.ID       Gold Silver Total
   <dbl> <fct>     <dbl> <chr>                                  <chr>              <int>  <int> <int>
1      1 A           100 A11, A09, B22, E12, A04, C09, D33, A01 A17                    8      1     9
2      1 B           200 A71, A09, B12, E32, A04, C19           B22, E12               6      2     8
3      2 C           300 B22, E12, A04, C09, D33, A01, M11, Z09 A44, C02, D03, A71     8      4    12
4      2 D           400 C09, D33                               A12, A01, M11, D18     2      4     6
user_34606700
Related