Is there a way to figure out how many gpu's on a node via slurm?

user_13614416 0
Viewed 746

I am working with a SLURM workload manager, and we have nodes with 4 GPUs.

The are several possible states of a node:

  • allocated (all computing resources are allocated)
  • mixed (part of the resources are allocated)
  • idle (none of the computing resources is allocated)

Canonical way to determine the resource utilization is sinfo command - https://slurm.schedmd.com/sinfo.html. There are several flags and options. However, it seems, that none of them can tell how many resources on a given node are allocated ath the moment.

Say, I would like to know, for the mixed state, whether 1,2 or 3 GPU's are available. Or this information is confidential and for some reason not available to users?

slurm server resources
1 Answers

If you run

scontrol show nodes

as a regular user, you will see a lot of information about the nodes, among which the line that look like

 AllocTRES=cpu=8,mem=48G,gres/gpu=2

tells you how many GPUs are allocated: gres/gpu=2. The other line

 CfgTRES=cpu=64,mem=257707M,billing=64,gres/gpu=2

tells how many GPUs are configured: gres/gpu=2. With these two lines, you can deduce the amount of GPUs still available on the node.

user_17636140
Related