SLURM: Cannot use more than 8 CPUs in a job

Question

Sorry for a stupid question. We setup a small cluster using slurm-16.05.9. The sinfo command shows:

NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
g01            1    batch*        idle   40   2:20:2 258120    63995      1   (null) none
g02            1    batch*        idle   40   2:20:2 103285    64379      1   (null) none
g03            1    batch*        idle   40   2:20:2 515734    64379      1   (null) none

So each node has 2 socket, each socket 20 cores, and totally 40 CPUs. However, we cannot submit a job using more than 8 CPUs. For example, with the following job description file:

#!/bin/bash
#SBATCH -J Test       # Job name
#SBATCH -p batch
#SBATCH --nodes=1
#SBATCH --tasks=1
#SBATCH --cpus-per-task=10
#SBATCH -o log
#SBATCH -e err

then submitting this job gave the following error message:

sbatch: error: Batch job submission failed: Requested node configuration is not available

even there is no jobs at all in the cluster, unless we set --cpus-per-task <= 8.

Our slurm.conf has the following contents:

ControlMachine=cbc1
ControlAddr=192.168.80.91
AuthType=auth/munge
CryptoType=crypto/munge
MpiDefault=pmi2
ProctrackType=proctrack/pgid
ReturnToService=0
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm-llnl/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
DefMemPerCPU=128000
FastSchedule=0
MaxMemPerCPU=128000
SchedulerType=sched/backfill
SchedulerPort=7321
SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory
AccountingStorageType=accounting_storage/none
AccountingStoreJobComment=YES
ClusterName=cluster
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log
SlurmdDebug=3
SlurmdLogFile=/var/log/slurm-llnl/slurmd.log
GresTypes=gpu

NodeName=g01 CPUs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=2 RealMemory=256000 State=UNKNOWN
NodeName=g02 CPUs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=2 RealMemory=256000 State=UNKNOWN Gres=gpu:P100:1
NodeName=g03 CPUs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=2 RealMemory=256000 State=UNKNOWN
PartitionName=batch Nodes=g0[1-3] Default=YES MaxTime=UNLIMITED State=UP

Could anyone give us a hint how to fixe this problem ?

Thank you very much.

T.H.Hsieh

user_1763614 · Answer

The problem is most probably not the number of CPUs but the memory. The job script does not specify memory requirements, and the configuration states

DefMemPerCPU=128000

Therefore the job with 10 CPUs is requesting a total of 1 280 000 MB of

The memory computation per job in that old version of Slurm is possibly related to cores and not threads so the actual request would be half of that 640 000 RAM on a single node while the maximum available is 515 734 MB.

A job requesting 8 CPUs under that hypothesis requests 512 000MB. You can confirm this with scontrol show job

SLURM: Cannot use more than 8 CPUs in a job

1 Answers