Why does slurm assign more tasks than I asked when I "sbatch" multiple jobs with a .sh file?

Viewed 710

I submit some cluster mode spark jobs which run just fine when I do it one by one with below sbatch specs.

#!/bin/bash -l

#SBATCH -J Spark
#SBATCH --time=0-05:00:00   # 5 hour
#SBATCH --partition=batch
#SBATCH --qos qos-batch
###SBATCH -N $NODES
###SBATCH --ntasks-per-node=$NTASKS
### -c, --cpus-per-task=<ncpus>
###     (multithreading) Request that ncpus be allocated per process
#SBATCH -c 7
#SBATCH --exclusive
#SBATCH --mem=0
#SBATCH --dependency=singleton

If I use a launcher to submit the same job with different node and task numbers, the system gets confused and tries to assign according to $SLURM_NTASK which gives 16. However I ask for example only 1 node,3tasks.

#!/bin/bash -l

for n in {1..4}
do
    for t in {3..4}
    do
        echo "Running benchmark with ${n} nodes and ${t} tasks per node" 
        sbatch -N ${n} --ntasks-per-node=${t} spark-teragen.sh
        sleep 5
        sbatch -N ${n} --ntasks-per-node=${t} spark-terasort.sh
        sleep 5
        sbatch -N ${n} --ntasks-per-node=${t} spark-teravalidate.sh
        sleep 5
    done
done

How can I fix the error below by preventing slurm assign weird number of tasks per node which exceeds the limit.

Error:

srun: Warning: can't honor --ntasks-per-node set to 3 which doesn't match the
requested tasks 16 with the number of requested nodes 1. Ignoring --ntasks-per-node.
srun: error: Unable to create step for job 233838: More processors requested than
permitted
0 Answers
Related