Foreach loop returns an error message for first set of workers only

Question

Something very strange started happening in a package I have been working on. I am using a foreach loop with a doparallel backend to perform repeated random walks with restarts on an adjacency matrix (reproducible example below).

install.packages("Matrix")
install.packages("foreach")
devtools::install_github("https://github.com/DavisWeaver/crosstalkr")

library(foreach)

m <- replicate(1000, sample(x = c(0,1), size = 1000, replace = TRUE))
w <- Matrix::Matrix(m, sparse = TRUE)
w <- Matrix::t(Matrix::t(w)/Matrix::colSums(w)) #normalize based on the column sum.
seeds <- sample(1:nrow(w), size = 32)

Each iteration should perform a random walk with restarts and return a numeric vector of affinity scores (so matrix in, one dimensional vector out). I tried to reproduce this error with a simpler task in each iteration but I wasn't able to. This setup worked fine for matrix::colSums(w) and several more trivial tasks.

The following error is returned for each of the first set of workers: "". Each subsequent iteration performs completely as expected with no errors. So when you run the first foreach loop below, you get the following image.

When you run subseqent foreach loops using the same parallel backend, each iteration performs as expected and there are no error messages!

cl <- parallel::makeCluster(4)
doParallel::registerDoParallel(cl)

n = 8
null_dist <-
  foreach::foreach(i = 1:n, .errorhandling = 'pass') %dopar% {
    crosstalkr::sparseRWR(w, seed_proteins = seeds, norm = FALSE)[[1]]
  }

null_dist <-
  foreach::foreach(i = 1:n, .errorhandling = 'pass') %dopar% {
    crosstalkr::sparseRWR(w, seed_proteins = seeds, norm = FALSE)[[1]]
  }


parallel::stopCluster(cl)

I am not sure where to go from here... I'm guessing it is something to do with how I am setting up the parallel backend? any advice would be appreciated.

user_1751961 · Answer

I'm not sure exactly why this is happening. However, to solve the problem you need to explicitly add .packages = "Matrix" to foreach.

Without specifying .packages, nrow doesn't produce the correct results.

Here is a simplified example

library(foreach)

w <- Matrix::Matrix()

You can see the first four values are incorrectly NULL.

cl <- parallel::makeCluster(4)
doParallel::registerDoParallel(cl)

null_dist <- foreach::foreach(i = seq_len(6)) %dopar% {
  nrow(w)
}

parallel::stopCluster(cl)

null_dist
#> [[1]]
#> NULL
#> 
#> [[2]]
#> NULL
#> 
#> [[3]]
#> NULL
#> 
#> [[4]]
#> NULL
#> 
#> [[5]]
#> [1] 1
#> 
#> [[6]]
#> [1] 1

Adding .packages = "Matrix" solves this problem.

cl <- parallel::makeCluster(4)
doParallel::registerDoParallel(cl)

null_dist <- foreach::foreach(i = seq_len(6), .packages = "Matrix") %dopar% {
  nrow(w)
}

parallel::stopCluster(cl)

null_dist
#> [[1]]
#> [1] 1
#> 
#> [[2]]
#> [1] 1
#> 
#> [[3]]
#> [1] 1
#> 
#> [[4]]
#> [1] 1
#> 
#> [[5]]
#> [1] 1
#> 
#> [[6]]
#> [1] 1

Foreach loop returns an error message for first set of workers only

1 Answers