'mc.cores' > 1 is not supported on Windows

user_13639668 0
Viewed 2373

I am new to programming and I am trying to use parallel processing for R in windows, using an existing code.

Following is the snippet of my code:


if (length(grep("linux", R.version$os)) == 1){
    num_cores = detectCores()
    impact_list <- mclapply(len_a, impact_func, mc.cores = (num_cores - 1))
  } 
  # else if(length(grep("mingw32", R.version$os)) == 1){
  #   num_cores = detectCores()
  #   impact_list <- mclapply(len_a, impact_func, mc.cores = (num_cores - 1))
  # 
  # }
  else{
    impact_list <- lapply(len_a, impact_func)
  }
  return(sum(unlist(impact_list, use.names = F)))

This works fine, I am using R on windows so the code enters in 'else' statement and it runs the code using lapply() and not by parallel processing.

I have added the 'else if' statement to make it work for windows. So when I un-comment 'else if' block of code and run it, I am getting an error "'mc.cores' > 1 is not supported on Windows". Please suggest how can I use parallel processing in windows, so that less time is taken to run the code. Any help will be appreciated.

multicore parallel-processing r windows
1 Answers

(disclaimer: I'm author of the future framework here)

The future.apply package provides parallel versions of R's built-in "apply" functions. It's cross platform, i.e. it works on Linux, macOS, and Windows. The package allows you to often just replace an existing lapply() with a future_lapply() call, e.g.

library(future.apply)
plan(multisession)

your_fcn <- function(len_a) {
  impact_list <- future_lapply(len_a, impact_func)
  sum(unlist(impact_list, use.names = FALSE))
}

Regarding mclapply() per se: If you use parallel::mclapply() in your code, make sure that there is always an option not to use it. The reason is that it is not guaranteed to work in all environment, that is, it might be unstable and crash R. In R-devel thread 'mclapply returns NULLs on MacOS when running GAM' (https://stat.ethz.ch/pipermail/r-devel/2020-April/079384.html), the author of mclapply() wrote on 2020-04-28:

Do NOT use mcparallel() in packages except as a non-default option that user can set for the reasons Henrik explained. Multicore is intended for HPC applications that need to use many cores for computing-heavy jobs, but it does not play well with RStudio and more importantly you don't know the resource available so only the user can tell you when it's safe to use. Multi-core machines are often shared so using all detected cores is a very bad idea. The user should be able to explicitly enable it, but it should not be enabled by default.

user_10720910
Related