In my bash script which is made to run on SLURM scheduler, I specified that I want jobs 1-900%10 to be run on the cluster, so 10 jobs in parallel will be found at any time.
I understand that based on 1-900%10 the first 10 must be run, then next 10, etc. Unless one of these finish before the other.
However I am finding out that SLURM is missing some jobs. Here is a screenshot:
Notice that jobs 5,6,7,8 are not there. I am not sure what the reason is, also they are never computed later on.
Here is a snippet of my bash script:
#!/usr/bin/env bash
#SBATCH --job-name=fotrcont
#SBATCH --account=anaccount
#SBATCH --partition=normal
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=16000
#SBATCH --mail-type=ALL
#SBATCH --mail-user=someone@example.com
#SBATCH --array=1-900%10
UPDATE
Turns out that for some reason the node the jobs are getting allocated to is drained for some reason. I have specified the node using nodelist as follows:
#!/usr/bin/env bash
#SBATCH --job-name=fotrcont
#SBATCH --account=anaccount
#SBATCH --partition=normal
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=16000
#SBATCH --mail-type=ALL
#SBATCH --mail-user=someone@example.com
#SBATCH --nodelist=onode09 # HERE
#SBATCH --array=1-900%10
However its now running 5 jobs at a time rather than 10, although I have specified 1-900%10. I have tried adding more nodes in --nodelistand increasing --node to 2 or 3 but it doesn't seem to work.