How to run jobs in paralell using one slurm batch script?

user_10868076 0
Viewed 2043

I am trying to run multiple python scripts in parallel with one Slurm batch script. Take a look at the example below:

#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=/dev/null
#SBATCH --error=/dev/null
#SBATCH --ntasks=2
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1G
#SBATCH --partition=All
#SBATCH --time=5:00

srun sleep 60
srun sleep 60
wait

How do I tweak the script such that the execution will take only 60 sec (instead of 120) ? Splitting the script into two scripts is not an option.

slurm parallel-processing
1 Answers

As written, that script is running two sleep commands in parallel, two times in a row.

Each srun command initiates a step, and since you set --ntasks=2 each step instantiates two tasks (here the sleep command).

If you want to run two 1-task steps in parallel, you should write it this way:

srun --exclusive -n 1 -c 1 sleep 60 &
srun --exclusive -n 1 -c 1 sleep 60 &
wait

Then each step only instantiates one task, and is backgrounded by the & delimiter, meaning the next srun can start immediately. The wait command makes sure the script terminates only when both steps are finished.

In that context, the xargs command and the GNU parallel commands can be useful to avoid writing multiple identical srun lines or avoiding a for-loop.

For instance, if you have multiple files you need to run your script over:

find /path/to/data/*.csv -print0 | xargs -0 -n1 -P $SLURM_NTASKS srun -n1 --exclusive python my_python_script.py

This is equivalent to writing as many

srun -n 1 -c 1 --exclusive python my_python_script.py /path/to/data/file1.csv &
srun -n 1 -c 1 --exclusive python my_python_script.py /path/to/data/file1.csv &
srun -n 1 -c 1 --exclusive python my_python_script.py /path/to/data/file1.csv &
[...]

GNU parallel is useful to iterate over parameter values:

parallel -P $SLURM_NTASKS srun  -n1 --exclusive python my_python_script.py ::: {1..1000}

will run

python my_python_script.py 1
python my_python_script.py 2
python my_python_script.py 3
...
python my_python_script.py 1000

Another approach is to just run

srun python my_python_script.py

and, inside the Python script, to look for the SLURM_PROCID environment variable and split the work according to its value. The srun command will start multiple instances of the script and each will 'see' a different value for SLURM_PROCID.

import os
print(os.environ['SLURM_PROCID'])
user_17636140
Related