Training the same model multiple times and save only the best one

Murilo 0
Viewed 46

I need to train n times the exactly same model (same hiperparameteres) with the same dataset for statistical assessment of performance. For that, i am using a for loop and i am using Keras ModelCheckpoint to save only the best model at each iteration.

Something like this:

n = 10
loss_train = []
loss_val = []

for i in range(n):
    model = Sequential(name=f'Conv1D_NE_{i}')
    model.add(InputLayer((timesteps, input_dim)))
    model.add(Conv1D(filters=256, kernel_size=8, activation='relu'))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Conv1D(filters=128, kernel_size=8, activation='relu'))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(units=96, activation='relu'))
    model.add(Dense(units=48, activation='relu'))
    model.add(Dense(units=24, activation='relu'))
    model.compile(optimizer='adam', loss='mean_squared_error')
    cp = ModelCheckpoint(
        filepath=f'C:/Users/.../Conv1D_{i}.hdf5',
        monitor='val_loss',
        verbose=1,
        save_best_only=True,
        mode='min')
    earlystop = EarlyStopping(monitor='val_loss',
                              patience=500,
                              verbose=1,
                              mode='min',
                              restore_best_weights=True)
    train = model.fit(x_train, y_train, batch_size=32,
                          epochs=2000, verbose=2,
                          callbacks=[cp, earlystop],
                          validation_split=0.2, shuffle=False)
    loss_train.append(min(train.history['loss']))
    loss_val.append(min(train.history['val_loss']))

I am saving the min loss during train and min val_loss during validation in separete lists. But i am unsure how to save only the best of those n models (regarding the min val_loss) for later use in my test data. Or if that's not possible, i could manually delete all but the best model, but is there anyway i can automate this process?

EDIT: Here's the solution using lists, to delete all the models except the best one:

 for i in range(len(loss_val)):
     if i != np.argmin(np.asarray(loss_val)):
         if os.path.exists(f'C:/Users/.../Conv1D_{i}.hdf5'):
             os.remove(f'C:/Users/.../Conv1D_{i}.hdf5')
         else:
             print('File does not exist.')
keras python
1 Answers

I think you need to create your own ModelCheckpoint callback, you only add the part of code which take in consideration the minimum of val_loss that you reach during your loop, the current model will be saved in case it generate smaller val_loss

class CMCP(tf.keras.callbacks.Callback):  # CMCP : Custom ModelCheckPoint
  def __init__(self, saving_path = None, best_model_loss = np.inf, verbose = 0):
    super(CMCP, self).__init__()
    self.saving_path = saving_path
    self.best_model_loss = best_model_loss
    self.verbose = verbose

  def on_epoch_end(self, epoch, logs=None):
    current_loss = logs.get("val_loss")
    if current_loss < self.best_model_loss:
      if self.verbose:
        print('best_val_loss improved from {:.5f} to {:.5f}, saving model to {}'.format(self.best_model_loss, current_loss, self.saving_path))
      self.best_model_loss = current_loss
      self.model.save(self.saving_path)

    else:
      if self.verbose:
        print('Epoch {:05}: best_val_loss did not improve from {:.5f}'.format(epoch, self.best_model_loss))


#implement customized model checkpoint callback
cmcp = CMCP(saving_path = saving_path, verbose = 1)

for i in range(n):
   ...
   ...
   train = model.fit(..., callbakcs = [..., cmcp])
   ...
Mohamed El Amine BELLEBNA0
Related