Where do I find slurm diagnostic information when a job just hangs?

user_8238447 0
Viewed 485

I am running slurm 20.11.8 on a system with 1 login node and 3 compute nodes. The status information I can find is below.

$ slurmd -V
slurm 20.11.8

$ sinfo -N
NODELIST       NODES PARTITION STATE 
pauli-node-01      1   normal* idle  
pauli-node-02      1   normal* idle  
pauli-node-03      1   normal* idle  

$ systemctl status slurmctld
● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-10-05 22:04:10 CDT; 10h ago
 Main PID: 11802 (slurmctld)
    Tasks: 7
   Memory: 6.7M
   CGroup: /system.slice/slurmctld.service
           └─11802 /usr/sbin/slurmctld -D

Oct 05 22:04:10 pauli.mer.utexas.edu systemd[1]: Started Slurm controller daemon.

Here is my configuration file:

$ cat /etc/slurm/slurm.conf

# slurm.conf file generated by configurator.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
SlurmctldHost=pauli
#SlurmctldHost=
#
#DisableRootJobs=NO
#EnforcePartLimits=NO
#Epilog=
#EpilogSlurmctld=
#FirstJobId=1
#MaxJobId=999999
#GresTypes=
#GroupUpdateForce=0
#GroupUpdateTime=600
#JobFileAppend=0
#JobRequeue=1
#JobSubmitPlugins=1
#KillOnBadExit=0
#LaunchType=launch/slurm
#Licenses=foo*4,bar
#MailProg=/bin/mail
#MaxJobCount=5000
#MaxStepCount=40000
#MaxTasksPerNode=128
MpiDefault=none
#MpiParams=ports=#-#
#PluginDir=
#PlugStackConfig=
#PrivateData=jobs
ProctrackType=proctrack/cgroup
#Prolog=
#PrologFlags=
#PrologSlurmctld=
#PropagatePrioProcess=0
#PropagateResourceLimits=
#PropagateResourceLimitsExcept=
#RebootProgram=
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
#SlurmdUser=root
#SrunEpilog=
#SrunProlog=
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
#TaskEpilog=
TaskPlugin=task/affinity
#TaskProlog=
#TopologyPlugin=topology/tree
#TmpFS=/tmp
#TrackWCKey=no
#TreeWidth=
#UnkillableStepProgram=
#UsePAM=0
#
#
# TIMERS
#BatchStartTimeout=10
#CompleteWait=0
#EpilogMsgTime=2000
#GetEnvTimeout=2
#HealthCheckInterval=0
#HealthCheckProgram=
InactiveLimit=0
KillWait=30
#MessageTimeout=10
#ResvOverRun=0
MinJobAge=300
#OverTimeLimit=0
SlurmctldTimeout=120
SlurmdTimeout=300
#UnkillableStepTimeout=60
#VSizeFactor=0
Waittime=0
#
#
# SCHEDULING
#DefMemPerCPU=0
#MaxMemPerCPU=0
#SchedulerTimeSlice=30
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU
#
#
# JOB PRIORITY
#PriorityFlags=
#PriorityType=priority/basic
#PriorityDecayHalfLife=
#PriorityCalcPeriod=
#PriorityFavorSmall=
#PriorityMaxAge=
#PriorityUsageResetPeriod=
#PriorityWeightAge=
#PriorityWeightFairshare=
#PriorityWeightJobSize=
#PriorityWeightPartition=
#PriorityWeightQOS=
#
#
# LOGGING AND ACCOUNTING
#AccountingStorageEnforce=0
AccountingStorageHost=pauli
#AccountingStoragePass=abcdef
#AccountingStoragePort=1234
AccountingStorageType=accounting_storage/none
AccountingStorageUser=slurm
AccountingStoreJobComment=YES
ClusterName=cluster
#DebugFlags=
JobCompHost=pauli
#JobCompLoc=slurm_comp_db
#JobCompPass=abcdef
#JobCompPort=1234
JobCompType=jobcomp/none
JobCompUser=slurm
#JobContainerType=job_container/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=info
#SlurmctldLogFile=
SlurmdDebug=info
#SlurmdLogFile=
#SlurmSchedLogFile=
#SlurmSchedLogLevel=
#
#

# POWER SAVE SUPPORT FOR IDLE NODES (optional)
#SuspendProgram=
#ResumeProgram=
#SuspendTimeout=
#ResumeTimeout=
#ResumeRate=
#SuspendExcNodes=
#SuspendExcParts=
#SuspendRate=
#SuspendTime=
#
#
# COMPUTE NODES
NodeName=pauli-node-0[1-2] CPUs=64 Sockets=2 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN
NodeName=pauli-node-03 CPUs=40 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN
PartitionName=normal Nodes=pauli-node-0[1-3] Default=YES MaxTime=INFINITE State=UP

When I try to run a simple job on 1 node with srun (along the lines of Slurm: Quick Start User Guide), the job just hangs. Does anyone know where I should look for diagnostic information to figure out why the job hangs?

$ srun -N1 -n1 -l hostname
slurm crash configuration
2 Answers

One of the first things to check is the network connectivity and making sure no firewall is in the way. You can check that with

scontrol ping

on the control nodes. Also, srun has a -v option that can tell you where it is blocked (you can add multiple of such options to increase the verbosity).

And of course, the log files for both the controller and the slurmd's may contain information. Again, the log level can be increased, with scontrol setdebug.

The usual suspects, besides the firewall, are often SELinux, netmasks, IP routes. Make also sure the clocks are in sync and munge is running OK.

user_17636140

SOLVED. The firewall for all compute nodes must be either "off" or configured to trust the other nodes in the system.

See Compute node firewall must be off

I was able to run (on Linux RedHat)

firewall-cmd --zone=trusted --add-source=10.xxx.xxx.xxx --add-source=10.xxx.xxx.xxx --add-source=10.xxx.xxx.xxx

on each compute node in order to avoid turning off the firewall altogether. I think the reason the problem came up recently is that the firewall was probably deactivated, and after a recent reboot of the system, the firewall came back up.

Thanks @damienfrancois for helping point me to firewall problems.

user_82384470
Related