I'm training a tensorflow 2.0 model on a training and validation set, and then want to later on recover the epoch with the lowest validation loss and evaluate it on the test set. I'm a bit confused on what is the best way to recover the model with the lowest validation loss after training. I solved this by doing the following:
- Add the following
ModelCheckpointcallback to my callbacks list
ModelCheckpoint(filepath=str(modelpath / 'best_model_{epoch:03d}-{val_loss:.6f}.h5'),
monitor='val_loss',
save_best_only=True)
- Implemented the following function:
# This function gets the path to the best model and cleans any non-best model
def get_best_model_path(modelpath):
modelpath_ = Path(modelpath)
best_model_list = [p for p in modelpath_.iterdir()]
best_model_list.sort()
for modelpath in best_model_list[:-1]:
os.remove(modelpath)
return best_model_list[-1]
- Then my code will be something like this
model.fit(<fit params>)
best_model_path = common.get_best_model(modelpath)
best_model = tf.keras.models.load_model(best_model_path)
evaluate(best_model, ...)
This works because ModelCheckpoint is set to save the best epoch only, so I know that the last epoch saved will be the one with (1) largest epoch number value and (2) best validation loss. Because of the way I save the file (using formater {epoch:03d}) I guarantee that it will end up at the end of the sort order when reading the file paths in get_best_model_path(modelpath)
While this works, it seems very cumbersome and I've been wondering if Tensorflow does not offer a better way to do this.