"These samplers cannot be used in parallelized code"

user_2072412 0
Viewed 93

I was reading the vignette for the rgen package which provides headers for sampling from some common distributions. In the first paragraph, it says that:

Please note, these samplers, just like the ones in armadillo cannot be used in parallelized code as the underlying generation routines rely upon R calls that are single-threaded.

This was news to me, and I've been using RcppArmadillo for quite some time now. I was wondering if someone could elaborate on this point (or provide references to where I can read about the issue). I'm especially interested in learning what "cannot be used" means here; will results be wrong, or will it just not parallelize?

rcpp r
1 Answers

These functions use R's random number generator, which must not be used in parallelized code, since that leads to undefined behavior. Undefined behavior can lead to virtually anything. From my point of view you are lucky if the program crashes, since this clearly tells you that something is going wrong.

The HPC task view lists some RNGs that are suitable for parallel computation. But you cannot use them easily with the distributions provided by rgen or RcppDist. Instead, one could do the following:

  • Copy function for multivariate normal distribution from rgen an adjust it's signature such that it takes a std::function<double()> as source for N(0, 1) distributed random numbers.
  • Use a fast RNG instead of R's RNG.
  • Use the same fast RNG in parallel mode.

In code as a quick hack:

// [[Rcpp::depends(RcppArmadillo)]]
#include <RcppArmadillo.h>
// [[Rcpp::plugins(cpp11)]]
// [[Rcpp::depends(dqrng)]]
#include <xoshiro.h>
#include <dqrng_distribution.h>
// [[Rcpp::plugins(openmp)]]
#include <omp.h>

inline arma::mat rmvnorm(unsigned int n, const arma::vec& mu, const arma::mat& S,
                         std::function<double()> rnorm = norm_rand){
  unsigned int ncols = S.n_cols;
  arma::mat Y(n, ncols);
  Y.imbue( rnorm ) ;
  return arma::repmat(mu, 1, n).t() + Y * arma::chol(S);
}

// [[Rcpp::export]]
arma::mat defaultRNG(unsigned int n, const arma::vec& mu, const arma::mat& S) {
  return rmvnorm(n, mu, S);
}

// [[Rcpp::export]]
arma::mat serial(unsigned int n, const arma::vec& mu, const arma::mat& S) {
  dqrng::normal_distribution dist(0.0, 1.0);
  dqrng::xoshiro256plus rng(42);
  return rmvnorm(n, mu, S, [&](){return dist(rng);});
}

// [[Rcpp::export]]
std::vector<arma::mat> parallel(unsigned int n, const arma::vec& mu, const arma::mat& S, unsigned int ncores = 1) {
  dqrng::normal_distribution dist(0.0, 1.0);
  dqrng::xoshiro256plus rng(42);
  std::vector<arma::mat> res(ncores);

  #pragma omp parallel num_threads(ncores)
  {
    dqrng::xoshiro256plus lrng(rng);      // make thread local copy of rng 
    lrng.jump(omp_get_thread_num() + 1);  // advance rng by 1 ... ncores jumps 
    res[omp_get_thread_num()] = rmvnorm(n, mu, S, [&](){return dist(lrng);});
  }
  return res;
}


/*** R
set.seed(42)
N <- 1000000
M <- 100
mu <- rnorm(M)
S <- matrix(rnorm(M*M), M, M)
S <- S %*% t(S)
system.time(defaultRNG(N, mu, S))
system.time(serial(N, mu, S))
system.time(parallel(N/2, mu, S, 2))
*/

Result:

> system.time(defaultRNG(N, mu, S))
   user  system elapsed 
  6.984   1.380   6.881 

> system.time(serial(N, mu, S))
   user  system elapsed 
  4.008   1.448   3.971 

> system.time(parallel(N/2, mu, S, 2))
   user  system elapsed 
  4.824   2.096   3.080

Here the real performance improvement comes from using a faster RNG, which is understandable since the focus here lies on many random numbers and not so much on matrix operations. If I shift more towards matrix operations by using N <- 100000 and M <- 1000 I get:

> system.time(defaultRNG(N, mu, S))
   user  system elapsed 
 16.740   1.768   9.725 

> system.time(serial(N, mu, S))
   user  system elapsed 
 13.792   1.864   6.792 

> system.time(parallel(N/2, mu, S, 2))
   user  system elapsed 
 14.112   3.900   5.859

Here we clearly see that in all cases user time is larger than elapsed time. The reason for this is the parallel BLAS implementation I am using (OpenBLAS). So there are quite a few factors to consider before deciding on a method.

user_84166100
Related