Output of foreach saves as a large file

Question

I am struggling with getting a reasonable size of the output of foreach loop in R when saved as .Rdata file. Basically, I fit several mixed effect models using different data sets, store the model outputs in a list and then save the list as .Rdata file.

In an illustrative example of my problem you can see that the sizes of the output objects (here mod_list and mod_list2) are the same when employing either for loop (sequential computation) or foreach loop (parallelized computation).

#sequential

library(lme4)
mod_list <- vector("list", length = 5)

for (i in 1:5) {
  #fit each model using a unique dataset
  m <- lmer(Reaction ~ Days + (1|Subject), sleepstudy[-i,])
  mod_list[i] <- list(m)
}

#parallel

library(doParallel)
library(foreach)

UseCores <- detectCores()
cl <- makeCluster(UseCores)
registerDoParallel(cl)

mod_list2 <- foreach(i=1:5) %dopar% {
  library(lme4)
  #fit each model using a unique dataset
  m <- lmer(Reaction ~ Days + (1|Subject), sleepstudy[-i,])
  m

}
stopCluster(cl)

Output object sizes:

print(object.size(mod_list), units = "auto")

143 Kb

print(object.size(mod_list2), units = "auto")

143 Kb

However, what differs is the size of the .Rdata files when saved (here 172 kB and 611 kB for mod_list and mod_list2, respectively).

Further, we can see that in for loop the models are fitted in the global R environment:

lapply(1:5, function (x) attributes(attributes(mod_list[[x]]@frame)$terms)$.Environment)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

while models coming from foreach loop are fitted each in a unique environment:

lapply(1:5, function (x) attributes(attributes(mod_list2[[x]]@frame)$terms)$.Environment)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

I suspect that this difference in environments between for and foreach loops is the reason of much larger .Rdata files in the latter case (in my real task they are six times larger). I have tried to use compression xz with various compression levels when saving the output but the reduction of the file size is not satisfactory.

My question is: Is there a way to reduce the output file size when using foreach loop?

Output of foreach saves as a large file

0 Answers