In Keras, there is an Adadelta optimiser for SGD as follows:
optimizer = optimizers.Adadelta(lr=1.0, rho=0.95, epsilon=None, decay=0.0)
Here is the doc.: https://keras.io/optimizers/#adadelta But as we know, Adadelta does not use any learning rate. So what is lr for?
It is indeed a very interesting question. Let's see how this is described in Keras documentation:
In the original version of Adadelta you don't have to set an initial learning rate. In this version, initial learning rate and decay factor can be set, as in most other Keras optimizers.
So the documentations itself admits that this method doesn't need a learning rate. I believe this design decision was made because of some other templates, dependencies, or codes in the project.
More specifically, the philosophy of keras is you can combine any building blocks you want (i.e. a unified API). If you remove the parameter lr from this, I believe you won't be able to use some of the callbacks.
Now, let's compare the Adadelta implementation of Keras to the original paper:
Line 406: here the gradients are accumulated into a moving average (a is the moving average, rho is decay rate as in the paper, g is computed gradients for parameter p):
new_a = self.rho * a + (1. - self.rho) * K.square(g)
self.updates.append(K.update(a, new_a))
This perfectly corresponds to the following line in the algorithm:
Line 410: delta calculation (here, d_a is delta accumulator, also in the form of moving average):
update = g * K.sqrt(d_a + self.epsilon) / K.sqrt(new_a + self.epsilon)
This perfectly corresponds to
Line 411: now here is the tricky part. The code looks as follows:
new_p = p - lr * update
Which doesn't follows the original algorithm in the paper:
Furthermore, such learning rate admits changes through the learning rate decay parameter. However, the default value of lr in Keras is 1.0, and decay is 0.0 so by default it shouldn't affect the outcome.