How to implement exponentially decay learning rate in Keras by following the global steps

user_8601361 0
Viewed 9080

Look at the following example

# encoding: utf-8
import numpy as np
import pandas as pd
import random
import math
from keras import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import Adam, RMSprop
from keras.callbacks import LearningRateScheduler

X = [i*0.05 for i in range(100)]

def step_decay(epoch):
    initial_lrate = 1.0
    drop = 0.5
    epochs_drop = 2.0
    lrate = initial_lrate * math.pow(drop, 
    math.floor((1+epoch)/epochs_drop))
    return lrate

def build_model():
    model = Sequential()
    model.add(Dense(32, input_shape=(1,), activation='relu'))
    model.add(Dense(1, activation='linear'))
    adam = Adam(lr=0.5)
    model.compile(loss='mse', optimizer=adam)
    return model

model = build_model()
lrate = LearningRateScheduler(step_decay)
callback_list = [lrate]

for ep in range(20):
    X_train = np.array(random.sample(X, 10))
    y_train = np.sin(X_train)
    X_train = np.reshape(X_train, (-1,1))
    y_train = np.reshape(y_train, (-1,1))
    model.fit(X_train, y_train, batch_size=2, callbacks=callback_list, 
              epochs=1, verbose=2)

In this example, the LearningRateSchedule does not change the learning rate at all because in each iteration of ep, epoch=1. Thus the learning rate is just const (1.0, according to step_decay). In fact, instead of setting epoch>1 directly, I have to do outer loop as shown in the example, and insider each loop, I just run 1 epoch. (This is the case when I implement deep reinforcement learning, instead of supervised learning).

My question is how to set an exponentially decay learning rate in my example and how to get the learning rate in each iteration of ep.

reinforcement-learning keras deep-learning neural-network
2 Answers

You can actually pass two arguments to the LearningRateScheduler. According to Keras documentation, the scheduler is

a function that takes an epoch index as input (integer, indexed from 0) and current learning rate and returns a new learning rate as output (float).

So, basically, simply replace your initial_lr with a function parameter, like so:

def step_decay(epoch, lr):
    # initial_lrate = 1.0 # no longer needed
    drop = 0.5
    epochs_drop = 2.0
    lrate = lr * math.pow(drop,math.floor((1+epoch)/epochs_drop))
    return lrate
user_36072030

The actual function you implement is not exponential decay (as you mention in your title) but a staircase function.

Also, you mention your learning rate does not change inside your loop. That's true because you set model.fit(..., epochs=1,...) and your epochs_drop = 2.0 at the same time. I am not sure this is your desired case or not. You are providing a toy example and it's not clear in that case.

I would like to add the more common case where you don't mix a for loop with fit() and just provide a different epochs parameter in your fit() function. In this case you have the following options:

  1. First of all keras provides a decaying functionality itself with the predefined optimizers. For example in your case Adam() the actual code is:

    lr = lr * (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))

which is not exactly exponential either and it's somehow different than tensorflow's one. Also, it's used only when decay > 0.0 as it's obvious.

  1. To follow the tensorflow convention of exponential decay you should implement:

    decayed_learning_rate = learning_rate * ^ (global_step / decay_steps)

Depending on your needs you could choose to implement a Callback subclass and define a function within it (see 3rd bullet below) or use LearningRateScheduler which is actually exactly this with some checking: a Callback subclass which updates the learning rate at each epoch end.

  1. If you want a finer handling of your learning rate policy (per batch for example) you would have to implement your subclass since as far as I know there is no implemented subclass for this task. The good part is that it's super easy:

Create a subclass

class LearningRateExponentialDecay(Callback):

and add the __init__() function which will initialize your instance with all needed parameters and also create a global_step variables to keep track of the iterations (batches):

   def __init__(self, init_learining_rate, decay_rate, decay_steps):
      self.init_learining_rate = init_learining_rate
      self.decay_rate = decay_rate
      self.decay_steps = decay_steps
      self.global_step = 0

Finally, add the actual function inside the class:

def on_batch_begin(self, batch, logs=None):
    actual_lr = float(K.get_value(self.model.optimizer.lr))
    decayed_learning_rate = actual_lr * self.decay_rate ^ (self.global_step / self.decay_steps)
    K.set_value(self.model.optimizer.lr, decayed_learning_rate)
    self.global_step += 1

The really cool part is the if you want the above subclass to update every epoch you could use on_epoch_begin(self, epoch, logs=None) which nicely has epoch as parameter to it's signature. This case is even easier as you could skip global step altogether (no need to keep track of it now unless you want a fancier way to apply your decay) and use epoch in it's place.

user_35847650
Related