Using doParallel with RMarkdown

Question

I am trying to use doParallel with RMarkdown. The R code calls a script called report.RMD. The goal is to produce 3 html reports from the iris dataset, each one named after the species with a table in that is filtered down to that species. The problem is that although the reports are rendered, the table doesn't get filtered to that species for example, the virginica.html document has the versicolor species listed. This appears to be a parallelisation issue because when %dopar% is changed to %do%, the html reports are produced as expected.

The wider objective is to use parallel processing with RMarkdown on a larger scale than on this example but am using the below just as an example.

The R code is:

library(doParallel)
library(tidyverse)

iris_list<-c("virginica","versicolor", "setosa")

no_cores <- detectCores() - 1  

cl <- makeCluster(no_cores)  
registerDoParallel(cl)  

foreach(i = 1:length(iris_list), .packages = (.packages())) %dopar% {

cat<-iris_list[i]
iris2<-iris%>%filter(Species==cat)
  
rmarkdown::render("report.RMD",
output_file = paste0(cat, ".html"))
}  

stopCluster(cl)

The RMD code is:

---
title: "Untitled"
date: "28 September 2021"
output: html_document
---



```{r table, echo=FALSE}
iris2

user_10021876 · Answer

Found one approach here: https://gist.github.com/hrbrmstr/17bc21af55392f23f012f57bb2fda51c#file-do_rpt-r

library(doParallel)
library(tidyverse)

# Define report parameters
reports <- list(list(out="setosa.html", params=list(sp="setosa")),
             list(out="virginica.html", params=list(sp="virginica")),
             list(out="versicolor.html", params=list(sp="versicolor")))

make_report <- function(r) {
  
  require(rmarkdown)
  
  tf <- tempfile()
  dir.create(tf)
  
  rmarkdown::render(input="report.RMD",
                    output_file=r$out,
                    intermediates_dir=tf,
                    params=r$params,
                    quiet=TRUE)
  unlink(tf)
  
}

no_cores <- detectCores() - 1  
cl <- makeCluster(no_cores)  
registerDoParallel(cl)  

foreach(r=reports, .combine=c) %dopar% make_report(r)

stopCluster(cl)

Rmarkdown file:

---
title: "Untitled"
date: "28 September 2021"
output: html_document
params:
  sp: "default"
---

```{r}
data(iris)
iris[iris$Species == params$sp,]

(plus three trailing ` for the r-chunk... stack formatting is confused).

Using doParallel with RMarkdown

1 Answers