Why does AdamOptimizer seem not apply the correct gradient?

Question

It's possible that I just misunderstood how Adam works, but why is this happening:

x = tf.Variable([0.0, 0.0]) # variable
y = tf.constant([5.0, 1.0]) # target
cost = tf.abs(x-y)**2

As the first dimension of y is larger than the second, the gradient in the first dimension is larger than the second (as it should be) and each dimension of x approaches its target value at its own rate:

sgd = tf.train.GradientDescentOptimizer(0.001)
train = sgd.minimize(cost)

with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for _ in range(5):
    sess.run(train)
    grad,variable = sess.run(opt.compute_gradients(cost))[0]
    print(grad,variable)

#[-9.98  -1.996] [0.01  0.002]
#[-9.96004  -1.992008] [0.01998  0.003996]
#[-9.94012  -1.988024] [0.02994004 0.00598801]
#[-9.920239  -1.9840479] [0.03988016 0.00797603]
#[-9.900399  -1.9800799] [0.0498004  0.00996008]

Why are the rates essentially equal if we use Adam, even though the gradients have quite different values?

adam = tf.train.AdamOptimizer(0.001)
train = adam.minimize(cost)

with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for _ in range(5):
    sess.run(train)
    grad,variable = sess.run(opt.compute_gradients(cost))[0]
    print(grad,variable)

#[-9.998 -1.998] [0.001 0.001]
#[-9.996 -1.996] [0.00199999 0.00199997]
#[-9.994     -1.9940002] [0.00299997 0.00299989]
#[-9.992001  -1.9920005] [0.00399994 0.00399976]
#[-9.99     -1.990001] [0.0049999  0.00499955]

user_5763590 · Answer

ADAM or adaptive momentum works as follows:

The velocity v accumulates the gradient Elements.

When you watch the Adam equations in this paper you will see, that the stepsize has an upper bound at the learning rate. In the paper they call this characteristics of Adam: "its careful choice of stepsize" (discussed in Section 2.1 of the paper). This is exactly what you observe here as "essential equal rates" during the first 5 steps, the rates in Adam are building up (accumulating) over multiple previous gradients, while the stepsize is being restricted to the learning rate itself.

On more Information on how the variable is calculated and updated in Tensorflow (see equations here).

Additional remarks on Adam:

The larger α is relative to the learning rate, the more previous gradients aﬀect the current direction.

In sgd, the size of the step was simply the norm of the gradient multiplied by the learning rate.

In Adam, the size of the step depends on how large and how aligned a sequence of gradients are. The step size is largest when many successive gradients point in exactly the same direction. If the momentum algorithm always observes Gradient g, then it will eventually accelerate in the direction of −g.

This is from the Deep Learning Book by Ian Goodfellow, in more Detail you can read about Adam here.

Why does AdamOptimizer seem not apply the correct gradient?

1 Answers