I have an Rscript which I wish to run as an array job via slurm on an HPC.
This is my slurm script:
#!/bin/bash --login
#SBATCH --job-name=JOBNAME
#SBATCH --export=ALL
#SBATCH --mail-user=MYEMAILADDRESS
#SBATCH --mail-type=BEGIN,END
#SBATCH --time=24:00:00
#SBATCH --output=%x-%A_%a.out
#SBATCH --array=1-20
#SBATCH --ntasks=1 --cpus-per-task=8
#SBATCH --mem=200G
#SBATCH --partition=compute
date
hostname
module load R-4.2.1
module list
Rscript script.R $SLURM_ARRAY_TASK_ID
and this is my Rscript
# /usr/bin/Rscript
args <- commandArgs(trailingOnly=TRUE)
task_id <- args[1]
a_list <- readRDS("/path/to/large/RDS")
# head(a_list) or rownames(a_list) etc all return "NULL" here
obj_subset <- a_list[[task_id]]
# do things with the subsetted element of list, e.g.
res <- function(obj_subset)
res_file <- paste0(task_id, "_results",".RDS")
saveRDS(res, res_file)
Reading in the RDS does not throw an error, however, if I try to print the contents of a_list or obj_subset into a log file, it returns NULL.
I'm at my wits end because my Rscript works in R console, but somehow readRDS() is not working (but also not returning an error) in an array job. I have tried both relative and absolute paths for the input RDS file. If it's relevant, the RDS input is on a NFS drive.
My suspicion is somehow multiple read connections to the same file is causing this but I can't find anything concrete to back this up, nor any idea on how to resolve this.
Thank you in advance!