Model parameter initialization

Question

I am new to PyTorch. I am writing a simple program for linear regression and i want to compare the results by using different methods (SGD,momentum,ADAM,etc). The problem I have is that I want every time a loop ends for the model parameters to be reinitialized to the same value that the previous model started with so the comparison is valid.

This is what i have so far, this is my training data:

      x1=np.arange(0,10,1).reshape(10,1)
      y1=2*x1+1+np.random.normal(0,1,(10,1))
      x=torch.from_numpy(x1)
      y=torch.from_numpy(y1)

Here i train the data

      from torch.utils.data import TensorDataset,DataLoader
      train=TensorDataset(xdata,ydata) 
      size_batch=10
      dl=DataLoader(train,size_batch,shuffle=True)

Define the model and opt

     model=nn.Linear(1,1)
     opt=torch.optim.SGD(model.parameters(), lr=1e-4)
     import torch.nn.functional as F
     loss1=F.mse_loss
     loss=loss1(model(x),y)

Function

     def fitmodel(nepochs, model, loss1, opt):
         for epoch in range(nepochs):
             for xm,ym in dl:
                 predict = model(xm)
                 loss = loss1(predict, ym)
                 loss.backward()
                 opt.step()
                 opt.zero_grad()

Call the function

     fitmodel(1000,model,loss1,opt)

Now i want to rerun the above but for different optimization algorithms. If I just rerun fitmodel it will use some of the parameters it already has calculated. I want to start from the same 'initial conditions' as i started the previous run. Anyone has any ideas how to do that?

Edit Before i run the fitmodel I copy the initial bias and weight

   w1=model.weight
   b1=model.bias
   fitmodel(1000,model,loss1,opt)
   model.weight=w1
   model.bias=b1
   loss=[]

But i get this error: TypeError: cannot assign 'list' as parameter 'bias' (torch.nn.Parameter or None expected)

user_4280242 · Answer

The parameters of a linear layer are stored in model.weight and model.bias. You need to copy those before training, and then restore afterwards. This is a bit more involved than what you're doing in your code. Example below

# clone and detach so that we have an actual backup copy,
# not merely a reference to the parameters
w1=model.weight.clone().detach()
b1=model.bias.clone().detach()

for i in range(3): # as many experiments as you wish to run

    # since we have detached, w1 and b1 are no longer nn.Parameter -
    # we have to rewrap them. We keep copying so that the tensors used
    # in the computation are separate from the backup copies
    model.weight=nn.Parameter(w1.clone())
    model.bias=nn.Parameter(b1.clone())

    # we reinitialize the optimizer because it looks at model.parameters()
    # if not for this line, it would try to optimize the values from 
    # the previous experiment!
    opt=torch.optim.SGD(model.parameters(), lr=1e-4)

    fitmodel(1000,model,loss1,opt)

Model parameter initialization

1 Answers