When does Pytorch initialize parameters?

user_15436955 0
Viewed 307

I’m now writing my own network with Pytorch. And I want to use a pretrained model in my net. Here is my overwriting init() code:

class Generator(nn.Module):
    def __init__(self) -> None:
        super(Generator, self).__init__()
        model_path = "somedir"
        chekpoint = torch.load(model_path)
        h_model = H_model()
        h_model.load_state_dict(chekpoint['model'])
        # 设置为测试模式
        h_model.eval()
        self.H_model = h_model
        self.unet = UNet(enc_chs=(9,64,128,256,512), dec_chs=(512, 256, 128, 64), num_class=3, retain_dim=False, out_sz=(304, 304))

Here, the h_model is loaded from checkpoint which I’ve trained it well. My question is that after the initialization, will the parameter in h_model changed(Are the pretrained parameters vaule being modified by some function?)? And why(I mean how does Pytorch treat self-defined layer when it initializes parameters? And when does Pytorch initialize parameters?)

pytorch initialization
1 Answers

For the basic layers (e.g., nn.Conv, nn.Linear, etc.) the parameters are initialized by the __init__ method of the layer.
For example, look at the source code of class _ConvNd(Module) (the class from which all other convolution layers are derived). At the bottom of its __init__ it calls self.reset_parameters() which initialize the weights.

Therefore, if your nn.Module does not have any "independent" nn.Parameters, only trainable parameters inside sub-nn.Modules, when you construct your network, all weights of the sub modules are being initialized as the sub modules are constructed. That is, once you call h_model = H_model() the weights of h_model are already initialized to their default values. Calling h_model.load_state_dict(...) overrides these values to the desired pre-trained weights.

user_17144100
Related