Currently, I can use srun [variety of settings] bash to create a shell on a compute note. However, if my ssh disconnects for whatever reason and I want to re-access the shell, how can I do that?
Currently, I can use srun [variety of settings] bash to create a shell on a compute note. However, if my ssh disconnects for whatever reason and I want to re-access the shell, how can I do that?
squeue -u $USERNAME
scontrol show job $JOBID | grep NodeList
sattach $JOBID.0
srun --pty --jobid $JOBID -w $NODEID /bin/bash
Assuming the SSH connection from your laptop to the login node of the cluster is unstable, you can use a terminal multiplexer such as screen or tmux, depending on what is already installed on the login node.
Typically, a session would look like this
[you@yourlaptop ~]$ ssh cluster-frontend
[you@cluster ~]$ tmux # to enter a persistent tmux session
[you@cluster ~]$ srun [...] bash # to get a shell on a compute node
[you@computenode ~]$ # some work, then...
some SSH error (e.g. Write failed: Broken pipe)
[you@yourlaptop ~]$ ssh cluster-frontend
[you@cluster ~]$ tmux a # to re-attach to the persistent tmux session
[you@computenode ~]$ # resume work
With screen, you would use screen -r rather than tmux a. Otherwise the process is the same.
If you want to join a job from another terminal instance (on the right below), you can use the Slurm's sattach command.
[you@yourlaptop ~]$ ssh cluster-frontend |
[you@cluster ~]$ srun [...] bash |
srun: job ******* queued and waiting for resources |
srun: job ******* has been allocated resources | [you@yourlaptop ~]$ ssh cluster-frontend
[you@computenode ~]$ | [you@cluster ~]$ sattach --pty ********
[you@computenode ~]$ echo OK | [you@computenode ~]$ echo OK
[you@computenode ~]$ OK | [you@computenode ~]$ OK
The original terminal and the one in which sattach was run are now entirely synchronised.
Note that the above does not protect from an accidental termination of srun; whenever srun terminates, the job is terminated as well.