How to train a deep learning model on a GPU server with laptop closed?

user_16711664 0
Viewed 180

I'm about to train my own ASR model using ESPNet on a GPU server. If my calculations are right, it's going to take about 4 consecutive days (using about 100G of audio data).

I'm mainly using VScode to remotely connect to the SSH server, and will run the shell file with the VScode terminal.

My question is that will I have to leave my laptop open for four days in order to train my model?

not sure if this is any useful info, but this is my nvcc --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

and my nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro RTX 6000     Off  | 00000000:00:06.0 Off |                    0 |
| N/A   32C    P0    41W / 250W |      0MiB / 22698MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Once my data is all prepared, I'll execute the run.sh file. Espnet github: https://github.com/espnet/espnet

The model I'm using is located in espnet/egs2/zeroth_korean/asr1.

I'm fairly new to linux servers and models this heavy and large, so any type of feedback would be much appreciated.

pytorch tensorflow linux python
1 Answers

Many Linux versions include the GNU Screen program, which - amongst other things - allow you to keep processes running after you've logged off.

Once connected, simply run the screen command:

[myhost ~]$ screen

Start your long running process inside this screen terminal.

You can now close the terminal. Power off, restart your computer, whatever.

When you want to check up on your process, just re-connect and run the following command to re-attach:

[myhost ~]$ screen -r

I hope this works for you.

screen has lots of other nice tricks. Just google "Linux Screen" for an abundance of articles on this.

user_21535880
Related