How should I understand warmup learning rate in tensorflow object detection api?

user_11758585 0
Viewed 2589

Could you explain how it works during training?

learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base: 8e-2
          total_steps: 300000
          warmup_learning_rate: .0001
          warmup_steps: 400
        }
      }```
object-detection tensorflow2.0 tensorflow
2 Answers

answering my own question :) With the setting above, training starts with lr=0.0001 and reaches to 0.08 at the end of 400 epochs (warmup_steps). Until 400th epoch lr is incremented linearly.

user_117585850

Why would you want to increase the learning rate over time? If I'm not wrong, it is better to first have a learning rate and then as we reach the optimum, it should be lesser and lesser as we do not want to cross the optimum.

user_65027610
Related