Does EarlyStopping take model of last epoch or last best score?

Question

I trained a model in keras using EarlyStopping in callbacks with patience=2:

model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=validation_split,
    callbacks= [keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=2, verbose=0, mode='min'), keras.callbacks.ModelCheckpoint('best_model.h5', monitor='val_loss', mode='min', verbose=1, save_best_only=True)], class_weight=class_weights)

Epoch 1/20
3974/3975 [============================>.] - ETA: 0s - loss: 0.3499 - accuracy: 0.7683
Epoch 00001: val_loss improved from inf to 0.30331, saving model to best_model.h5
3975/3975 [==============================] - 15s 4ms/step - loss: 0.3499 - accuracy: 0.7683 - val_loss: 0.3033 - val_accuracy: 0.8134
Epoch 2/20
3962/3975 [============================>.] - ETA: 0s - loss: 0.2821 - accuracy: 0.8041
Epoch 00002: val_loss improved from 0.30331 to 0.25108, saving model to best_model.h5
3975/3975 [==============================] - 14s 4ms/step - loss: 0.2819 - accuracy: 0.8043 - val_loss: 0.2511 - val_accuracy: 0.8342
Epoch 3/20
3970/3975 [============================>.] - ETA: 0s - loss: 0.2645 - accuracy: 0.8157
Epoch 00003: val_loss did not improve from 0.25108
3975/3975 [==============================] - 14s 4ms/step - loss: 0.2645 - accuracy: 0.8157 - val_loss: 0.2687 - val_accuracy: 0.8338
Epoch 4/20
3962/3975 [============================>.] - ETA: 0s - loss: 0.2553 - accuracy: 0.8223
Epoch 00004: val_loss did not improve from 0.25108
3975/3975 [==============================] - 15s 4ms/step - loss: 0.2553 - accuracy: 0.8224 - val_loss: 0.2836 - val_accuracy: 0.8336
Wall time: 58.4 s

Obviously the model didn't improve after epoch 2, but patience=2 led the algorithm terminate after 4 epochs. When I run model.evaluate now, does it take the model trained after 4 epochs or does it take the model trained after 2 epochs? Is there any need to save and load the best model with ModelCheckpoint then in order to evaluate?

user_3653343 · Answer

In you specific case, it retain the model after 4 epochs:

as I can see, the model did indeed improve after the first 2 epochs:

accuracy
0.7683 - 0.8043 - 0.8157 - 0.8224

loss
0.3499 - 0.2821 - 0.2645 - 0.2553

Early stopping would stop your training after the first two epochs if had there been no improvements. In this situation your loss isn't stuck after the first two epochs

So early stopping is not called at the end of epoch 2, and the model returned is the one at the end of your first 4 epochs and so the training has continued

From tensorflow:

Assuming the goal of a training is to minimize the loss. With this, the metric to be monitored would be 'loss', and mode would be 'min'. A model.fit() training loop will check at end of every epoch whether the loss is no longer decreasing, considering the min_delta and patience if applicable. Once it's found no longer decreasing, model.stop_training is marked True and the training terminates.

If your model had not improved for after two epochs, your returned model would have been the model after the first two epochs.

The pointed parameter restore_best_weights is not directly connected to the early stopping.

It simply restore the weights of the model at the best detected performances. Always from tf:

Whether to restore model weights from the epoch with the best value of the monitored quantity. If False, the model weights obtained at the last step of training are used.

To make an example, you can use restore_best_weights even without early_stopping. The training would end only at the end of your set number of epochs, but the model returned at the end would have been the one with the best performances. This could be useful in the situation of bouncing loss due to a wrong learning rate and\or optimizer

refs: early stopping tf doc

Does EarlyStopping take model of last epoch or last best score?

1 Answers