I use google colab for training my model. I use tensorflow 2.2.
Here is how I create summary writer:
writer = tf.summary.create_file_writer(os.path.join(model_dir, 'logs'), max_queue=1)
Here is my code which I run each step:
with writer.as_default():
tf.summary.scalar("train_loss", total_loss, step=num_steps)
writer.flush()
The problem is that if model_dir is just /content/model_dir, then everything saves fine, but if I save my model to a folder on google drive (I connect to my google drive with this code:
from google.colab import drive
drive.mount('/content/gdrive2', force_remount=True)
) then event file doesn't get updated. It is being created, but it is not filling with data during training (and even after training is finished).
As I understand, the problem is that google drive doesn't understand that tensorflow updates the event file. But after training is finished, the whole event file is saved. What can I do to fix this bug?