Batch script for multi-partition job?

Viewed 3600

I'm working on a project which runs programs on two different partitions of a large compute cluster. I'd like to run this using a batch script, but after searching, it's still unclear if/how I can allocate and run programs on two different partitions from within a single batch script. Here's the sort of thing I'd like to do

#!/bin/bash
#SBATCH --partition=<WHAT GOES HERE? I want to perform 100 processes on partition "batch" and 1 process on partition "gpu". I will alternate between the 2 during my jobs execution>
#SBATCH --ntasks=<100 on batch, 1 on gpu>
#SBATCH --mem-per-cpu=2G
#SBATCH --time=4-00:00:00
#SBATCH --exclude=nodeynode[003,016,019,020-023,026-030,004-015,017-018,020,024,031]
#SBATCH --job-name="lorem_ipsum"

filenames=("name1" "name2" "name3")

srun -p gpu python gpu_init.py
wait

for i in {0..100}
do
    for name in "${filenames[@]}"
    do
    srun -p batch pythonexecutable &
    done
srun -p gpu python gpu_iter.py
wait
done

Apologies for bash errors, I usually script in python but I can't here as I'm switching python modules (different versions) within my bash script (not shown). I saw that you can actually put a list of partitions in the header of a batch script, but from what I read that actually just tells the scheduler to allocate any available partitions from within the list, not multiple partitions.

Thanks!

1 Answers
Related