"rejoin" a bash SLURM job

Viewed 1199

Currently, I can use srun [variety of settings] bash to create a shell on a compute note. However, if my ssh disconnects for whatever reason and I want to re-access the shell, how can I do that?

2 Answers

Get your job ID

squeue -u $USERNAME

Get your NodeID

scontrol show job $JOBID | grep NodeList

Attach (you can assign step = 0)

sattach $JOBID.0

This did not work for me, but in theory this should work:

srun --pty --jobid $JOBID -w $NODEID /bin/bash

Source

Assuming the SSH connection from your laptop to the login node of the cluster is unstable, you can use a terminal multiplexer such as screen or tmux, depending on what is already installed on the login node.

Typically, a session would look like this

   [you@yourlaptop ~]$ ssh cluster-frontend
   [you@cluster ~]$ tmux # to enter a persistent tmux session
   [you@cluster ~]$ srun [...] bash # to get a shell on a compute node
   [you@computenode ~]$ # some work, then...
   some SSH error (e.g. Write failed: Broken pipe)
   [you@yourlaptop ~]$ ssh cluster-frontend
   [you@cluster ~]$ tmux a # to re-attach to the persistent tmux session
   [you@computenode ~]$ # resume work

With screen, you would use screen -r rather than tmux a. Otherwise the process is the same.

If you want to join a job from another terminal instance (on the right below), you can use the Slurm's sattach command.

[you@yourlaptop ~]$ ssh cluster-frontend           |  
[you@cluster ~]$ srun [...] bash                   |
srun: job ******* queued and waiting for resources |
srun: job ******* has been allocated resources     | [you@yourlaptop ~]$ ssh cluster-frontend
[you@computenode ~]$                               | [you@cluster ~]$ sattach --pty ********
[you@computenode ~]$ echo OK                       | [you@computenode ~]$ echo OK
[you@computenode ~]$ OK                            | [you@computenode ~]$ OK

The original terminal and the one in which sattach was run are now entirely synchronised.

Note that the above does not protect from an accidental termination of srun; whenever srun terminates, the job is terminated as well.

Related