how to submit slurm job array with different input files

user_10208724 0
Viewed 1000

I have a list of text files (~ 200 files) that needs to be processed. So I was trying to submit a slurm job array for this task but I cannot find a solution. What I have tried is to submit a multiple jobs (~ 200 jobs) by looping through the files for just a single task. I am sure that there is a way to create job array for this problem, could you please advice me?

That is my bash script

#!/bin/bash
input_list=$1
prefix=$2

readarray list < $input_list
a=0
for i in "${list[@]}"
do
   file=$i
   sbatch -a1-1 -- $PWD/kscript.sh $file $prefix"_"$a
   a=`expr $a + 1`
done
slurm
1 Answers

I figured the solution so it is simpler than I thought, here it is:

#!/bin/bash
# Task name
#SBATCH -J myjob
# Run time limit
#SBATCH --time=4:00:00
# Standard and error output in different files
#SBATCH -o %j_%N.out.log
#SBATCH -e %j_%N.err.log
#SBATCH --ntasks=1
# Execute application code

input_list=$1
prefix=$2

readarray list < $input_list
file=${list[$SLURM_ARRAY_TASK_ID-1]}
input=$PWD"/"$file
output=$prefix"_"$SLURM_ARRAY_TASK_ID
./kscript.sh $input $output
user_102087240
Related