I am training an NLP model using HuggingFace Trainer's API. When doing training with
trainer.train()
it prints out the loss after N training steps (I can specify the logging steps in the TrainingArguments class). What I've noticed is that if I plot the training losses printed to the console, there is a remarkable drop at the start of each epoch, and then the loss flattens out for the rest of the epoch. Picture below
Is this happening because the printed loss is being averaged across each epoch (meaning that the last logging step of each epoch prints the average loss of the whole epoch and not just the loss for that logging step)? If this is not the case, what are other possible explanations for this behaviour?