Reconnecting to jupyter notebook loses output

user_850781 0
Viewed 1100

I have a remote server with a long-running jupyter notebook process (keras training). When I reconnect to the notebook (either by reloading the chrome tab or by clicking on the notebook in the list of running notebooks), I get a tab marked as "running" (hourglass icon instead of the book icon), and I can confirm that the process is running by ssh to the server. However, the progress indicator and stdout of the process running is lost. The bar

71255/225127 [========>.....................] - ETA: 3:32:43 - loss: 2.1890

is never updated. Other (non-keras) processes lose their stdout (unless I also write to a file).

Is this a known problem?

Is there a way to recover the output stream after reconnecting?

jupyter jupyter-notebook
1 Answers

This is a known problem, and as far as I know IPython has already published a collaboration jupyter notebook option, that they said "could solve the problem in the collaboration mode"

Sadly, if it's lost, there's no way to find it back (unless you outputted to a file or to a variable, or you know how to do some intricate tricks that could recover the data)

One way to solve this, to prevent the issue in advance, is to use magic code %%capture <VARIABLE NAME> in the cell you want to save the stdout when closed the notebook in your browser (don't terminate your notebook). In this way, output will be save in <VARIABLE NAME> and later on you can access it using print(<VARIABLE NAME>.stdout), after the process is finished.

user_120171650
Related