After torch.load the performance decreases dramatically and after evaluate it increases

I got a weird bug. This is my model.

class MyModel(nn.Module):
    def __init__(self, feat_dim, num_classes):
        super(MyModel, self).__init__()
        self.model_resnet = models.resnet50(pretrained=False)
        num_ftrs = self.model_resnet.fc.in_features
        self.model_resnet.fc = nn.Identity()
        
        self.head1 = nn.Sequential(
                nn.Linear(num_ftrs, num_ftrs),
                nn.ReLU(inplace=True),
                nn.Linear(num_ftrs, feat_dim)
            )
        
        self.head2 = nn.Linear(num_ftrs, num_classes)
        
    def forward(self, x):
        self.eps=self.eps+1
        x = self.model_resnet(x)
        feat = F.normalize(self.head1(x), dim=1)
        classes = self.head2(x)
        
        return feat,classes
        

The following code is for saving and loading

torch.save(model.state_dict(),"./test.pth")
model.load_state_dict(torch.load("test.pth"))

Then I trained it and saved weights with test accuracy 0.95. Next time I load it and test something. It is like random guessing and accuracy is near to 0.

After I evaluate it with the whole test set. The test accuracy return to 0.8 but still lose performance.

I checked model.state_dict() the weights are the same before and after evaluating the whole test set.

Anyone has any ideas?