Suppose I want to run a program with 100 different input arguments. This is what I would do on my laptop for example:
for i in {1..32}
do
./test.sh $i
done
where test.sh is just a dummy program
#!/bin/bash
name=$(hostname)
touch $1.test
echo $name >> $1.test
echo $name
sleep 5
if I run the program it'll take approx. 160sec. I try to run a Slurm job on a cluster 40 nodes with 4 cpus each with my test.slurm:
#! /bin/bash
#
#SBATCH --ntasks=4
start=`date +%s`
for i in {1..32}
do
srun -l -n1 -N1 -c1 ./test.sh $i &
done
wait
end=`date +%s`
runtime=$((end-start))
echo $runtime
i get a runtime of 190 seconds instead of the expected ~40 so no multiprocesing but if i specify 2 nodes with --nodes=2 or more than 4 tasks ie a second node has to be allocated i get down to 90 seconds