TensorBoard does not update in real time

Viewed 751

I'm trying to understand how to use TensorBoard for model debugging, used the example from TF docs (https://www.tensorflow.org/tensorboard/scalars_and_keras#training_the_model_and_logging_loss), but my scalars don't update.

# multi-class classification with Keras
import tensorflow as tf
import datetime as dt
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold


log_dir = "/home/bartek/Desktop/Kaggle/trees/logs/scalars/" + 
dt.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir)

def NN_model():
    # create model
    model = Sequential()
    model.add(Dense(len(features.columns)*2, input_dim=len(features.columns), activation='relu'))
    model.add(Dense(len(features.columns)*2, activation='relu'))
    model.add(Dense(len(features.columns)*2, activation='relu'))
    model.add(Dense(len(features.columns)*2, activation='relu'))
    model.add(Dense(len(features.columns)*2, activation='relu'))
    model.add(Dense(len(features.columns)*2, activation='relu'))
    model.add(Dense(84, activation='relu'))
    model.add(Dense(49, activation='relu'))
    model.add(Dense(7, activation='softmax'))
    # Compile model
    model.compile(loss='categorical_crossentropy', 
                  optimizer='adam', 
                  metrics=[tf.keras.metrics.CategoricalAccuracy(),tf.keras.metrics.Precision(), tf.keras.metrics.Recall()], 
                  callbacks=[tensorboard_callback])
    return model

classifier = KerasClassifier(build_fn=NN_model, epochs=100, batch_size=5, verbose=0)
classifier.fit(features.values, labels.values, verbose=1, callbacks=[tensorboard_callback])

but all I get is terminal output:

enter image description here

and TensorBoard shows nothing:

enter image description here

Why this is the case? Manual refresh doesn't work either. Sometimes graphs stop after xth epoch and don't refresh.

1 Answers

Try to use the workaround suggested in here by adding profile_batch=0 to tf.keras.callbacks.TensorBoard.

ts_board = tf.keras.callbacks.TensorBoard(
            log_dir=log_dir,
            histogram_freq=1,
            update_freq='epoch',
            profile_batch=0
        )
Related