Slurm - randomize GPU allocation

user_2924735 0
Viewed 147

We are setting up Slurm on a single computer-server equipped with 4 GPUs. I have been trying to find a way to have Slurm allocate a job to a random GPU, where I use "random" in a relaxed way, meaning for instance that if GPU 1 is being used, then a new job requesting GPU resources should be allocated on a random GPU anomg those that are free (0, 2, 3)

The problem is that, as it is now, Slurm always allocates GPU0 first, then GPU1, then 2, 3. This causes GPU0 to execute considerably more jobs over time, and therefore wear down faster. A randomized selection would be better.

I have read the documentation regarding Sbatch etc, but it seems that one can select a specific GPU to run a job, but not pick it randomly as described above.

slurm
1 Answers

I face the same problem and wrote a program doing this. Please see https://github.com/gqqnbig/randomize-gpu

Mainly, you run the following script before starting slurmd.

#! /usr/bin/zsh -e

setopt extended_glob
# --force bypasses the error
# "rm: missing operand", when there are no files to delete.
rm --force /dev/rNvidia[0-9]#(N)
nvidias=( $(printf '%s\n' /dev/nvidia[0-9]# | shuf) )
for ((i = 1; i <= $#nvidias; i++)) do
    echo "Link "${nvidias[$i]}" to rNvidia$((i-1))"
    ln -s ${nvidias[$i]} /dev/rNvidia$((i-1))
done
user_7464610
Related