Keras callback ReduceLROnPlateau - cooldown parameter

user_9328846 0
Viewed 5045

ReduceLROnPlateau callback in Keras seems to be an interesting tool to use in training models. But I could not really figure out exactly what the cooldown parameter means in the callback function ReduceLROnPlateau in Keras.

Here is what the documentation says:

First, the interface of the function:

keras.callbacks.ReduceLROnPlateau(monitor='val_loss', 
                                  factor=0.1, 
                                  patience=10, 
                                  verbose=0, 
                                  mode='auto', 
                                  min_delta=0.0001, 
                                  cooldown=0, 
                                  min_lr=0)

ReduceLROnPlateau: Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced.

cooldown: number of epochs to wait before resuming normal operation after lr has been reduced.

The explanation does not really make it clear to me. Is it meant here that: - Say that lr=A. And the learning rate is reduced if the relevant monitored metric does not improve during patience number of epochs. (And say that lr=B after reducing it.) - And the learning rate is set to its first value (lr=A again) after cooldown number of epochs.

Is my understanding correct? If not, what is the real function of cooldown parameter here?

PS. When I google it, I see some examples where people set the cooldown parameter to zero, which makes me think that my perception on this parameter is wrong.

keras tensorflow machine-learning callback
1 Answers

True, it does not state it clearly in the description. What it means is that if you set a cooldown you have to wait before resuming normal operation (i.e. beginning to monitor if there is any improvement in the monitored metric over a patience epochs).

For example, let's say cooldown=5. After the learning rate is reduced, the algorithm waits 5 epochs before starting to monitor the metrics again. So if there is no improvement in the metric and patience=10, the learning rate will be reduced again after 15 epochs.

You can confirm this by looking at the corresponding code.

user_56687100
Related