How can I add code between the inner and outer loops of nested foreach loops in R

user_6182825 0
Viewed 378

I've read that the correct way to do nested foreach loops in R is via the nesting operator %:% (e.g. https://cran.r-project.org/web/packages/foreach/vignettes/nested.html).

However, code can't be added between the inner and outer loops when using this approach -- see example below.

Is there a way to create nested, parallelised foreach loops such that code can be added between the inner and outer loops?

More generally, is there anything wrong with the obvious way that springs to mind, namely simply nesting two foreach loops with the %dopar% operator instead of the %:% operator? See trivial example below.

library(foreach)

# Set up backend
cl = makeCluster(6)
registerDoParallel(cl)
on.exit(stopCluster(cl))

# Run nested loop with '%:%' operator. Breaks if adding code between the inner and outer loops 
foreach(i=1:2) %:% 
  # a = 1 #trivial example of running code between outer and inner loop -- throws error 
  foreach(j = 1:3) %dopar% {
    i * j
  }

# Run nested loop using 2 '%dopar%' statements -- is there anything wrong with this?
foreach(i=1:2, .packages = 'foreach') %dopar% {
  a = 1 #trivial example of running code between outer and inner loop
  foreach(j = 1:3) %dopar% {
    i * j
  }
}
doparallel parallel-processing r foreach
1 Answers

The chapter "Using %:% with %dopar%" from documentation you provided gives a useful hint:

all of the tasks are completely independent of each other, and so they can all be executed in parallel

The %:% operator turns multiple foreach loops into a single loop. That is why there is only one %do% operator in the example above. And when we parallelize that nested foreach loop by changing the %do% into a %dopar%, we are creating a single stream of tasks that can all be executed in parallel.

When you combine two %dopar% and measure execution time, you see that only the outer loop is executed in parallel, this is probably not what you're looking for :

system.time(
foreach(i=1:2, .packages = 'foreach') %dopar% {
  # Outer calculation
  Sys.sleep(.5)
  foreach(j = 1:3) %dopar% {
    # Inner calculation
    Sys.sleep(1)
  }
})
#  user      system     elapsed 
#  0.00        0.00        3.52

This elapsed time reflects:

parallel[ outer(0.5s) + sequential [3 * inner(1s)] ] ~ 3.5s

If the outer calculation is not too long, putting it into the inner loop is actually faster because the 6 workers of your example are used:

system.time(res <- foreach(i=1:2, .packages = 'foreach') %:%
  foreach(j = 1:3) %dopar% {
    # Outer calculation
    Sys.sleep(.5)
    # Inner calculation
    Sys.sleep(1)
  })
#  user      system     elapsed 
#  0.02        0.02        1.52

If the outer calculation is too long and you have much more inner loops that outer loops, you could precalculate the outer loop in parallel. You can then use the result within %:%:

system.time({
  precalc <- foreach(i=1:2) %dopar% {
    # Outer pre-calculation
    Sys.sleep(2)
    i
  }
  foreach(i=1:2, .packages = 'foreach') %:%
    foreach(j = 1:12) %dopar% {
      # Inner calculation
      Sys.sleep(1)
      precalc[[i]]*j
    }
})
#   user  system elapsed 
#   0.11    0.00    5.25

Is faster than :

system.time({
  foreach(i=1:2, .packages = 'foreach') %:%
    foreach(j = 1:12) %dopar% {
      # Outer calculation
      Sys.sleep(2)
      
      # Inner calculation
      Sys.sleep(1)
      i*j
    }
})

#   user  system elapsed 
#   0.13    0.00    9.21
user_135133280
Related