I was reading an article called Differential Learning Rates on Medium and was wondering if this could be applied in Keras. I was able to find this technique implemented in pytorch. Is this possible to implement in Keras? Is there any example code that someone could provide?
1 Answers
Starting from Tensorflow 2.4, there is an argument in tf.keras.optimizers.Optimizer: gradient_transformers. It is a list of functions that take and return a list of pairs [(gradient, Variable), ...]. You can implement differential learning rates as transformations that decrease learning rates depending on a weight position in the network via this parameter.
Related