Slurm node with 32 CPUs only runs 16 jobs at most

user_3994698 0
Viewed 35

I have setup a slurm queue with 2 nodes. Each node has 32 CPUS. But they both only run a maximum of 16 jobs at a time. The relevant lines in slurm.conf are below.

SelectTypeParameters=CR_CPU
NodeName=node1 NodeAddr=x.x.x.x CPUs=32 RealMemory=125953 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN
NodeName=node0 NodeAddr=x.x.x.x CPUs=32 RealMemory=127297 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN

I read online that "SelectTypeParameters=CR_CPU" should allocate jobs per CPU instead of core because we have 16 cores and 2 CPUs per core. But that doesn't seem to solve the problem. I also tried removing CUPs, CoresPerSocket, and ThreadsPerCore in the next lines. But slurmd would not start in that case and saying the configuration doesn't match hardware. Evidentally, without those parameters specified, it would take default which is 1.

Does anyone know where else I should be looking to fix this problem? The goal is to be able to run 32 jobs on each node at the same time.

slurm
1 Answers

Try removing CoresPerSocket=16 ThreadsPerCore=2 and leaving CPUs=32 only. IIRC in this case Slurm does not verify anything and just assumes there are 32 slots available.

user_17636140
Related