I have a model written in R. This model seems to require 32 GB RAM. I want to run the model 200 times with AWS-EC2. Initially I planned to use a c5.12xlarge Spot Instance. The c5.12xlarge instance has 48 vCPUs and 96 memory (GiB). At first I thought I would be able to run the model 48 times simultaneously because of the 48 vCPUs. But now I am thinking I might only be able to run the model three times simultaneously because 32 x 3 = 96 matching the 96 memory (GiB).
If I submit 200 models at once will the instance immediately run out of memory and cause every model to crash or be killed? Is there a way I can manage the memory used per model to prevent every model from crashing?
Here is the R code I use to implement parallel processing. Can this code be modified to assign 32 GiB memory per model?
setwd('/home/ubuntu/')
library(doParallel)
detectCores()
my.AWS.n.cores <- detectCores()
registerDoParallel(my.cluster <- makeCluster(my.AWS.n.cores))
folderName <- 'model000222b'
files <- list.files(folderName, full.names=TRUE)
start.time <- Sys.time()
foreach(file = files, .errorhandling = "remove") %dopar% {
source(file)
}
stopCluster(my.cluster)
end.time <- Sys.time()
total.time.c <- end.time-start.time
total.time.c
Should I simply modify the R code to only allow 3 cores maybe using:
my.AWS.n.cores <- 3
Or will AWS-EC manage the memory behind the scenes so all 200 models are able to run without me specifying anything about number of cores or memory to be reserved per model?
If I can only use 3 cores at once then perhaps I should select a difference instance type that has fewer vCPUs and more GiB of memory? Perhaps I should even consider using an x2iedn.4xlarge instance with 16 vCPUs and 512 GiB memory?
Thank you for any suggestions on how to run this model 200 times efficiently on AWS-EC2 given its large memory requirement.