How to continue printing the Intermediate results in jupyter after the reconnection?

user_16917734 0
Viewed 51

Today, I used jupyter to run a deep learning model remotely.
After the browser was disconnected for some time, I reconnected the running kernel, but jupyter did not continue to print the intermediate output results.
From the usage of GPU and the command line of jupyter, we can see that the kernel continues to run.
Is there any way I can continue to observe the intermediate output of the kernel?

the situation of the running kernel

jupyter-notebook python
1 Answers

The Google colab lifetime with the open browser is usually 12 hours.

The best way to save your changes is to use Checkpoint for your deep learning model to avoid losing the last trained model.

This is an example of how you can use checkpoint callback in your deep learning model while training, more examples and details can be found here.

# Include the epoch in the file name (uses `str.format`)
checkpoint_path = "training_2/cp-{epoch:04d}.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

batch_size = 32

# Create a callback that saves the model's weights every 5 epochs
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path, 
    verbose=1, 
    save_weights_only=True,
    save_freq=5*batch_size)

# Create a new model instance
model = create_model()

# Save the weights using the `checkpoint_path` format
model.save_weights(checkpoint_path.format(epoch=0))

# Train the model with the new callback
model.fit(train_images, 
          train_labels,
          epochs=50, 
          batch_size=batch_size, 
          callbacks=[cp_callback],
          validation_data=(test_images, test_labels),
          verbose=0)
user_151482000
Related