ModelCheckpoint saves data at a wrong place

Question

I was doing deep learning to classify fashion MNIST data on Google Colab, so train_images, train_labels refer to the training images and training labels from the fashion MNIST data.

Note that I have run the following code beforehand so that the Colab notebook can access my google drive:

from google.colab import drive
drive.mount('/content/gdrive')

First, I trained the following model and saved its checkpoint and weights to the folder /content/gdrive/My Drive/Colab/checkpoints/:

model1 = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])

model1.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

CHECKPOINT_PATH = '/content/gdrive/My Drive/Colab/checkpoints/'
checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=CHECKPOINT_PATH,
    save_weights_only=True,
    monitor='accuracy',
    mode='max',
    save_best_only=True)

model1.fit(train_images, train_labels, epochs=5, callbacks=[checkpoint_callback])

This worked fine; after model1 was fitted, I saw the checkpoint and weights file in the CHECKPOINT_PATH directory.

Then I decided I wanted to save new models in subdirectories of CHECKPOINT_PATH instead, so I wrote the following function that was intended to create a subdirectory of CHECKPOINT_PATH called name and save the checkpoint and weights of a model in the subdirectory.

def checkpoint_callback(name):
  CHECKPOINT_PATH = '/content/gdrive/My Drive/Colab/checkpoints/'
  path = os.path.join(CHECKPOINT_PATH, name)
  os.mkdir(path) # if file already exists, will raise FileExistsError
  callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=path, 
    save_weights_only=True,
    monitor='accuracy',
    mode='max',
    save_best_only=True)
  return callback

Then I tested this function on model3:

model3 = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])

model3.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

model3.fit(train_images, train_labels, epochs=5, callbacks=[checkpoint_callback('model3')])

After running the above, in the CHECKPOINT_PATH folder, I did see the model3 folder, but it was empty, and model3's checkpoint and weight data all appear in the CHECKPOINT_PATH folder. More specifically, when I ran the following in Colab

cd gdrive/My Drive/Colab/checkpoints/
ls

the output was checkpoint model3/ model3.data-00000-of-00001 model3.index.

How do I get model3's data, i.e. checkpoint, model3.data-00000-of-00001, and model3.index, to be saved in the model3 folder as I intended?

ModelCheckpoint saves data at a wrong place

0 Answers