Jupyter notebook : kernel restarts suddenly

user_14758339 0
Viewed 3225

I use Jupyter lab and Jupyter Notebook for my Deep Learning programs, so I made some long runs in order to train my models. But for some weeks, I've had recurrent kernel restarts after hours of training, which is very annoying. In addition, very few informations are given by the server console or by the browser log :

Jupyter-lab server log :

[I 2021-02-26 00:40:03.756 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
kernel 1330ee40-a826-44e2-9be9-f123deeaa1b2 restarted
[I 2021-02-26 00:40:04.070 ServerApp] Starting buffering for 1330ee40-a826-44e2-9be9-f123deeaa1b2:1b7fa111-f2d2-4804-bd90-c81e26562254
[I 2021-02-26 00:40:04.112 ServerApp] Restoring connection for 1330ee40-a826-44e2-9be9-f123deeaa1b2:1b7fa111-f2d2-4804-bd90-c81e26562254

I have the same problem when I use Jupyter-notebook instead of Jupyter-lab.

Various remarks :

  • The server and the client are not on the same machine, therefore I use ssh to connect to the server as described here.
  • I work under a corporation proxy
  • I use Tensorflow 2 for Deep Learning
tensorflow2.0 jupyter-lab jupyter-notebook python-3.x
3 Answers
user_134127930

Ok I thing I found the error's cause -> It was certainly a little memory leak in the code I was running which caused program crash after hundreds of epochs.

user_147583390

Somehow related: I have also seen this issue when using CUDA-DL libraries in Jupyter Notebooks created from Kubeflow:

AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports

In this case, it is due to a lack of memory assigned to the pod Kubeflow creates to deploy the Jupyter Server. The solution is as simple as allocating more memory (from ~4Gi) when creating the Notebook (and avoiding memory leaks in code :) )

user_189379770
Related