If I have multiple Tensorboard files, how can they be combined into a single Tensorboard file?
Say in keras the following model.fit() was called multiple times for a single model, for example in a typical GAN implementation:
for i in range(num_epochs):
model.fit(epochs=1, callbacks=Tensorboard())
This will produce a new Tensorboard file each time, which is not useful. Not sure if there is way to have Tensorboard append, or not produce unique time-stamped files each callback call.
It seems in tensorboard 2.3, you can access the tensorboard file's logged data and load it into a pandas dataframe. This tutorial outlines the approach:
https://www.tensorflow.org/tensorboard/dataframe_api
At the time of writing, I couldn't find a tensorboard version 2.3, but the module the tutorial relies upon - tensorboard.data.experimental.ExperimentFromDev() - seems to be present in tensorboard 2.2: https://github.com/tensorflow/tensorboard/blob/master/tensorboard/data/experimental/experiment_from_dev.py#L71
You could load existing data from several tensorboard files into dataframes and then combine those dataframes in the desired manner to write the combined dataframe to a new tensorboard file. I had considered this approach to address the problem of having multiple tensorboard files for different training runs but haven't tried it yet in my project.
If anyone stumbles upon this question, you can use initial_epoch defined in model.fit()
Inside the for loop, initial_epoch can be changed every time model.fit() is called.
New link for the initial_epoch documentation: https://keras.io/api/models/model_training_apis/
Also note that epochs will also need to be changed.
for i in range(num_epochs):
model.fit(epochs=i, initial_epoch=i-1, callbacks=Tensorboard())
because
Note that in conjunction with initial_epoch, epochs is to be understood as "final epoch". The model is not trained for a number of iterations given by epochs, but merely until the epoch of index epochs is reached.