I pretrained a model in my own local dataset. And then translating to a new situation with a bigger dataset(same domain). As the well-trained model is robust and I just simply preprocess the data and some environment settings to be the same then use a very low learning rate 5e-5 to finetune the weight-loaded network.
Here is my scheduler and the optimizer I am using:
self.optimizer = torch.optim.SGD(self.network.parameters(), self.initial_lr, weight_decay=self.weight_decay, momentum=0.99, nesterov=True)
and the learning rate adjustment is for poly_lr() updating like this.
self.optimizer.param_groups[0]['lr'] = poly_lr(current_epoch, self.max_num_epochs, self.initial_lr, 0.9)
And after about 20 epochs(I run 150 iterations each epoch), the loss decreased slow visually, but maybe not accurate. I have no idea if something goes wrong.
Cause in my own experience, the loss while finetuning, that won't decrease very fast but can be traced the tendency with the slow-decrease until convergence. Compare with training a new-begin model from scratch, the loss will drop very fast and then slowly go steady.
Is such an understanding correct?