I am using the fastLink package to identify duplicates in a large dataset. Normally, I can use fastLink::getMatches() to return a dataset with a variable that identifies the duplicate observations.
To speed up runtime, I am also using fastLink::blockData() to run the linking algorithm on individual blocks and then aggregating the blocks using fastLink::aggregateEM(). This returns an object with class fastLink.agg. The getMatches function does not appear to work with this class. How can I extract the matches for data that has been blocked and then aggregated?
Reprex:
library(fastLink)
library(foreach)
data <- data.frame(gender = c(1,2,1,1,1,1,2,2,1,2),
age = c(18, 25, 18, 35, 45, 55, 65, 76, 87, 98))
blocks <- blockData(data, data, varnames = c("gender"))
tmp_clus <- parallel::makeCluster(spec = parallel::detectCores()-2,
type = 'PSOCK')
doParallel::registerDoParallel(tmp_clus)
em_list <- foreach::foreach(i = 1:length(blocks), .verbose = F) %dopar%
{
library(fastLink)
data_block <- data[blocks[[i]]$dfA.inds,]
fastLink(
dfA = data_block, dfB = data_block,
varnames = c("gender", "age")
)
}
parallel::stopCluster(tmp_clus)
em_aggregated <- aggregateEM(em_list)
data_dedupe <- getMatches(dfA = data, dfB = data,
fl.out = em_aggregated)
# Error in getMatches(dfA = data, dfB = data, fl.out = em_aggregated) :
# dfA and dfB are identical, but fl.out is not of class 'fastLink.dedupe.' Please check your inputs.