I'm using a self supervised pretrained ViT-B with frozen weights (pretrained on ImageNet). I was just training a 2 layer MLP on ~ 16k images from a completely different domain for 120 epochs. These 2 layers are whithin the "model.head" module from the timm ViT model: My training loss decreases in the first 10 epochs to a minimum of 1.7ish and then increases again to eventually hover around 2 for the rest of the training (I'm using CrossEntropyLoss). Tried some hyperparamter tuning (increasing/decreasing Ir, weight decay, Ir scheduler, batch size), but my minimum loss went only from 2 to perhaps 1.7, which is still a dumb log loss (wild guesses). My accuracy starts at over 90 percent though and stays that way...?
Now, doing transfer learning from ImageNet to images from a completely different domain was perhaps not a very good idea. I'm looking to unfreeze the backbone next and train for 100 epochs or so. I did that already for a few epochs but the loss shot up to 3 while hovering around there (i stopped the training early on).
Any suggestions as to why l'm terribly failing at this?