Tensorflow Adam optimizer vs Keras Adam optimizer

user_4031331 0
Viewed 3776

I originally developed a classifier in Keras, where my optimizer was very easy to apply decay to.

adam = keras.optimizers.Adam(decay=0.001)

Recently I tried to change the entire code to pure Tensorflow, and cannot figure out how to correctly apply the same decay mechanism to my optimizer.

optimizer = tf.train.AdamOptimizer()
train_op = optimizer.minimize(loss=loss,global_step=tf.train.get_global_step())

How do I apply the same learning rate decay seen in my Keras code snippet to my Tensorflow snippet?

keras tensorflow python
2 Answers

You can find a decent documentation about decay in tensorflow:

...
global_step = tf.Variable(0, trainable=False)
starter_learning_rate = 0.1
learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step,
                                           100000, 0.96, staircase=True)

learning_step = ( tf.train.GradientDescentOptimizer(learning_rate)
    .minimize(...my loss..., global_step=global_step)
)

tf.train.exponential_decay applies exponential decay to the learning rate.

Other decays:

Keras implemented decay in AdamOptimizer similar to below, which is very close to inverse_time_decay in tensorflow:

lr = self.lr * (1. / (1. + self.decay * self.iterations))
user_14627700

You can find some useful hints of what you wanted to do here https://machinelearningmastery.com/understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks/.

To answer your question, I quote this source :

The callbacks operate separately from the optimization algorithm, although they adjust the learning rate used by the optimization algorithm. It is recommended to use the SGD when using a learning rate schedule callback

Base on this article you will find how to use the keras.callbacks and hopefully success to set the learning rate of Adam keras optimizer as you wished. Though, note that this is not recommended (I haven't tried it yet)

user_82266460
Related