Parallelise a for loop inside an R-script using slurm

Question

I have thousands of data frames and I want to parallelise their analysis into slurm. Here I am providing a simplified example:

I have an Rscript that I call: test.R

test.R contains these commands:

library(tidyverse)

df1 <- tibble(col1=c(1,2,3),col2=c(4,5,6))
df2 <- tibble(col1=c(7,8,9),col2=c(10,11,12))
files <- list(df1,df2)


for(i in 1:length(files)){
  df3 <- as.data.frame(files[1]) %>% 
    summarise(across(everything(), list(mean=mean,sd=sd)))
  
  write.table(df3, paste0("df",i))
}

^{Created on 2022-04-15 by the reprex package (v2.0.1)}

I want to parallelise the for loop and analyse each data frame as a separate job. Any help, guidance, and tutorials are appreciated.

Would the array command help?

#!/bin/bash
### Job name
#SBATCH --job-name=parallel
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --time=10:00:00
#SBATCH --mem=16G
#SBATCH --array=1-2

module load R/4.1.3
Rscript test.R $SLURM_ARRAY_TASK_ID

user_18524196 · Answer

If the dataframes are stored in separate files with numerical identifiers, you can create a list of these files as a bash variable and submit the array of identifying numbers in your sbatch command. The $SLURM_ARRAY_TASK_ID can be used as an input argument for your R code to point to that specific dataframe's file.

Just for the purpose of the following example, say the dataframes are stored as .csv files with names dataframe_1.csv, dataframe_2.csv, ... dataframe_100. Your command to run the parallel jobs would be something like this:

sbatch -a 1-100 jobscript.sh

And jobscript.sh would resemble your sample code in your question:

#!/bin/bash
### Job name
#SBATCH --job-name=parallel
#SBATCH --nodes=1
#SBATCH --time=10:00:00
#SBATCH --mem=16G

module load R/4.1.3
Rscript test.R "dataframe_${SLURM_ARRAY_TASK_ID}.csv"

Note that you may need to break up these sbatch jobs to allow for processing thousands of dataframes. If a file name has four digits in its numeric ID, you can prepend the first digit before the ${SLURM_ARRAY_TASK_ID} in the last line of the jobscript.sh:

Rscript test.R "dataframe_1${SLURM_ARRAY_TASK_ID}.csv"

or if you want to avoid making multiple scripts, pass that number as an argument to jobscript.sh:

sbatch -a 1-100 jobscript.sh 1

#!/bin/bash
### Job name
#SBATCH --job-name=parallel
#SBATCH --nodes=1
#SBATCH --time=10:00:00
#SBATCH --mem=16G

prepend=$1
module load R/4.1.3
Rscript test.R "dataframe_${prepend}${SLURM_ARRAY_TASK_ID}.csv"

Parallelise a for loop inside an R-script using slurm

1 Answers