I'm reading a paper where the authors has following setting. $Learning Rate= 1^{-3}, momentum=0.9, \beta_2=0.01, \lambda=1, \epsilon=10^{-4}$
However, tensorflow has following setting for Adam.
tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False,
name='Adam', **kwargs)
My question is, how do I set $momentum$ and $\lambda$
You could have a look in this in the documentation:
opt = tf.keras.optimizers.Adam(learning_rate=0.1)
var1 = tf.Variable(10.0)
loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1
step_count = opt.minimize(loss, [var1]).numpy()
# The first step is `-learning_rate*sign(grad)`
var1.numpy()
Beta1 and beta 2 : are the momentum decay; please check this:https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam
https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam