Hparams plugin with tf.keras (tensorflow 2.0)

The trick is to define the Hparams config with the path in which TensorBoard saves its validation logs.

So, if your TensorBoard callback is set up as:

log_dir = 'path/to/training-logs'
tensorboard_cb = TensorBoard(log_dir=log_dir)

Then you should set up Hparams like this:

hparams_dir = os.path.join(log_dir, 'validation')

with tf.summary.create_file_writer(hparams_dir).as_default():
    hp.hparams_config(
        hparams=HPARAMS,
        metrics=[hp.Metric('epoch_accuracy')]  # metric saved by tensorboard_cb
    )

hparams_cb = hp.KerasCallback(
    writer=hparams_dir,
    hparams=HPARAMS
)

I managed but not entirely sure what was the magic word. Here my flow in case it helps.

callbacks.append(hp.KerasCallback(log_dir, hparams))

HP_NUM_LATENT = hp.HParam('num_latent_dim', hp.Discrete([2, 5, 100])) 
hparams = {
   HP_NUM_LATENT: num_latent,
}

model = create_simple_model(latent_dim=hparams[HP_NUM_LATENT])  # returns compiled model
model.fit(x, y, validation_data=validation_data, 
          epochs=4,
          verbose=2,
          callbacks=callbacks) 

Since I have lost a couple of hours because of this. I would like to add to the good remark of Julian about defining the hparams config, that the tag of the metric you like to log with hparams and possibly its group in hp.Metric(tag='epoch_accuracy', group='validation') should match the one of a metric that you capture with Keras model.fit(..., metrics=). See hparams_demo for a good example

I just want to add to the previous answers. If you are using TensorBoard in a notebook on Colab, the issue may not be due to your code, but due to how TensorBoard is run on Colab. And the solution is to kill the existing TensorBoard and launch it again.

Please correct me if I am wrong.

Sample code:

from tensorboard.plugins.hparams import api as hp

HP_LR = hp.HParam('learning_rate', hp.Discrete([1e-4, 5e-4, 1e-3]))
HPARAMS = [HP_LR]
# this METRICS does not seem to have any effects in my example as 
# hp uses epoch_accuracy and epoch_loss for both training and validation anyway.
METRICS = [hp.Metric('epoch_accuracy', group="validation", display_name='val_accuracy')]
# save the configuration
log_dir = '/content/logs/hparam_tuning'
with tf.summary.create_file_writer(log_dir).as_default():
  hp.hparams_config(hparams=HPARAMS, metrics=METRICS)


def fitness_func(hparams, seed):
  rng = random.Random(seed)

  # here we build the model
  model = tf.keras.Sequential(...)
  model.compile(..., metrics=['accuracy'])  # need to pass the metric of interest

  # set up callbacks
  _log_dir = os.path.join(log_dir, seed)
  tb_callbacks = tf.keras.callbacks.TensorBoard(_log_dir)  # log metrics
  hp_callbacks = hp.KerasCallback(_log_dir, hparams)  # log hparams

  # fit the model
  history = model.fit(
    ..., validation_data=(x_te, y_te), callbacks=[tb_callbacks, hp_callbacks])


rng = random.Random(0)
session_index = 0
# random search
num_session_groups = 4
sessions_per_group = 2
for group_index in range(num_session_groups):
  hparams = {h: h.domain.sample_uniform(rng) for h in HPARAMS}
  hparams_string = str(hparams)
  for repeat_index in range(sessions_per_group):
    session_id = str(session_index)
    session_index += 1
    fitness_func(hparams, session_id)

To check if there is any existing TensorBoard process, run the following in Colab:

!ps ax | grep tensorboard

Assume PID for the TensorBoard process is 5315. Then,

!kill 5315

and run

# of course, replace the dir below with your log_dir
%tensorboard --logdir='/content/logs/hparam_tuning'

In my case, after I reset TensorBoard as above, it can properly log the metrics specified in model.compile, i.e., accuracies.