I am trying to count the number of addresses a customer by removing fuzzy duplicates.
Data sample:
address <- c("UNIT 3412 MANANSALA TOWER ESTRELLA ST ROCKWELL POBLACION MAKATI CITY",
"UNIT 3412 MANANSALA ESTRELLA ST BRGY POBLACION ROCKWELL . MAKATI CITY 1210",
"LOT 1 AND 3 WESTWOOD SUBD MANDURRIAO ILOILO CITY ILOILO",
"M VELEZ ST, NA, GUADALUPE, CEBU CITY, CEBU, PH",
"1410 CAVITE ST, TONDO I / II, NCR, CITY OF MANILA, FIRST DISTRICT")
customer_id <- c(1,1,1,2,2)
dt <- data.table(cbind(address,customer_id))
Finding exact duplicates is straightforward:
dt[, .(addresses = length(unique(address))), by = customer_id]
customer_id addresses
1: 1 3
2: 2 2
Problem with exact matching is that it will consider the 1st 2 addresses of customer_id 1 different when they are really the same.
So my question is: Is there a fuzzy equivalent to function unique()?
Result I expect:
customer_id addresses
1: 1 2
2: 2 2
I tried using find_duplicates from the revtools package to do this:
find_duplicates(dt, match_variable = "address", group_variables = "customer_id", method = "osa")
but it is really slow and the results are not great:
[1] 1 2 3 4 5
[1] "address"
attr(,"group_variables")
[1] "customer_id"
attr(,"match_function")
[1] "stringdist"
attr(,"method")
[1] "osa"
attr(,"threshold")
[1] 5