how to stop training if the sparse_validation_accuracy is higher than a baseline?

user_1609428 0
Viewed 242

I am fitting a huggingface model and trying to set up an early stopping when the sparse_validation_accuracy is better than 95%.

I am using the following call:

early_stopper = tf.keras.callbacks.EarlyStopping(monitor='accuracy', 
                                                 baseline = 0.90,
                                                 patience = 0,
                                                 restore_best_weights=True)

# train the model 
model.fit(train_dataset.shuffle(len(x_train)).batch(BATCH_SIZE),
          epochs=N_EPOCHS,
          batch_size=BATCH_SIZE,
          callbacks = [early_stopper])

Unfortunately, the model keeps training as shown below. Am I missing something?

0.9688WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.6875s vs `on_train_batch_end` time: 1.1250s). Check your callbacks.
  73/7495 [..............................] - ETA: 3:43:56 - loss: 0.1147 - sparse_categorical_accuracy: 0.9546
tensorflow python
3 Answers

Actually, built-in EarlyStopping callback works only upon epoch end. Thus, it won't stop your training in the middle of an epoch. If you want a callback, that would stop training while epoch isn't over yet, try creating your custom callback as an inheritor of tf.keras.callbacks.Callback. You will need to override the on_train_batch_end method.

Your resulting callback may look this way:

class CustomEarlyStopping(tf.keras.callbacks.Callback):
    def on_train_batch_end(self, logs=None):
        if logs['sparse_categorical_accuracy'] > 0.95:
            self.model.stop_training = True

I haven't practiced TF for quite a while, so this code might not work out of the box, but it's something to start with. More info can be found in official docs on writing custom callback and Callback class reference.

user_105688830

The Early Stopping callback does not just stop merely because a given monitored quantity has exceeded the baseline.

Instead, training will stop once, after reaching the baseline, the quantity does not improve any further.

In addition, this check is only done at the end of an epoch (at least according to the tensorflow documentation) so in your case, you're still in the middle of the epoch and thus training will continue.

If you want a hard stop once a given accuracy has been achieved at the end of a given batch I think you'll have to write your own callback. Shouldn't be too hard with all the guides out there.

You can use this here as a guide: https://www.tensorflow.org/guide/keras/custom_callback#early_stopping_at_minimum_loss

user_3811790

validation metrics are only calculated at the end of an epoch, so you can not stop training on a batch by evaluation the validation metric. However you can stop training at the end of an epoch. Below is the code for a custom callback that will stop training at the end of an epoch when the quantity being monitored reaches or exceeds the float value threshold. In the callback parameter monitor can be set to either 'train' or 'valid' If monitor='train' then training will halt if the training accuracy meets or exceeds the threshold value. If monitor='valid' then training will halt when the validation accuracy meets or exceeds the threshold value. In the callback, parameter model is the name of your compiled model. If the threshold level is met or exceeded during training your_model weights are set to the weights for the epoch on which the threshold was exceeded. A message is also printed out on that epoch stating training has been halted and the value of the metric being monitored is printed out.

class HALT(keras.callbacks.Callback):
    def __init__(self, model,  threshold, monitor):
        super(HALT, self).__init__()
        self.model=model        
        self.threshold=threshold
        self.monitor=monitor
        
    def on_epoch_end(self,epoch, logs=None):
        v_acc=logs.get('val_accuracy')        
        if self.monitor == 'train':
            metric=logs.get('accuracy')
        else:
            metric=logs.get('val_accuracy')  
        if metric >= threshold:
            weights=self.model.get_weights()
            self.model.set_weights(weights)
            print( f'\nthe threshold of {threshold} was reached on epoch {epoch} with the monitored value of {metric}', flush=True)
            self.model.stop_training = True # stop training

to use the callback the code is below

model=your_model # the variable name of your compiled model
threshold=.95 # set the desired threshold level
monitor='valid' # monitor the validation accuracy
callbacks=[HALT(model=model, threshold=threshold, monitor=monitor)]

In your_model.fit() include callbacks=callbacks.

user_107989170
Related