I am loading a pretrained model for transfer learning. The model has an encoder and a decoder. I want to replace the pretrained decoder with a smaller one. I tried loading the pretrained model into the smaller (incompatible one). I, of course, got the IncompatibleKeys error showing the unexpected keys (There are no missing keys since the new model is smaller).
Is there a better way to do what I described above? The new model seems to work fine.