How can I get matches from a fastLink.agg object from the fastLink package in R?

Viewed 24

I am using the fastLink package to identify duplicates in a large dataset. Normally, I can use fastLink::getMatches() to return a dataset with a variable that identifies the duplicate observations.

To speed up runtime, I am also using fastLink::blockData() to run the linking algorithm on individual blocks and then aggregating the blocks using fastLink::aggregateEM(). This returns an object with class fastLink.agg. The getMatches function does not appear to work with this class. How can I extract the matches for data that has been blocked and then aggregated?

Reprex:

library(fastLink)
library(foreach)

data <- data.frame(gender = c(1,2,1,1,1,1,2,2,1,2),
                   age = c(18, 25, 18, 35, 45, 55, 65, 76, 87, 98))

blocks <- blockData(data, data, varnames = c("gender"))

tmp_clus <- parallel::makeCluster(spec = parallel::detectCores()-2, 
                                  type = 'PSOCK')  
doParallel::registerDoParallel(tmp_clus)

em_list <- foreach::foreach(i = 1:length(blocks), .verbose = F) %dopar%
  {
    library(fastLink)
    data_block <- data[blocks[[i]]$dfA.inds,]
    
    fastLink(
      dfA = data_block, dfB = data_block, 
      varnames = c("gender", "age")
      )
  }
parallel::stopCluster(tmp_clus)

em_aggregated <- aggregateEM(em_list)

data_dedupe <- getMatches(dfA = data, dfB = data,
                          fl.out = em_aggregated)

# Error in getMatches(dfA = data, dfB = data, fl.out = em_aggregated) : 
#   dfA and dfB are identical, but fl.out is not of class 'fastLink.dedupe.' Please check your inputs.

0 Answers
Related