I was running a convnet model using Keras with tensorflow backend on Google Cloud, using the tensorboard callback to save a tfevents log for the training history. When I was monitoring the learning curve I noticed that half way through the training (learning curve was on plateau), a new tfevents log was saved to disk. And TensorBoard's learning curve graph showed that the training was reset to epoch #1 with val_loss also reset to scratch.
This is really weird. Does anyone know what is going on here? Under what circumstances would Keras automatically restart the training and save a new tfevents log?