Slightly adapt L1 loss to a weighted L1 loss in Pytorch, does gradient computation still work properly?

user_3383318 0
Viewed 1744

I implemented a neural network in Pytorch and I would like to use a weighted L1 loss function to train the network.

The implementation with the regular L1 loss contains this code for each epoch:

optimiser.zero_grad()
net.train()        
_,forecast = net(torch.tensor(feature, dtype=torch.float).to(DEVICE))
loss = F.l1_loss(forecast, torch.tensor(target,dtype=torch.float).to(DEVICE),reduction='mean')
loss.backward()
params.append(net.parameters())
optimiser.step()

Now I want to use a weighted L1 loss instead. So I thought to use the same standard Pytorch L1 function again and rescale the forecasts and targets with weights. Will the gradient computation still be done correctly?

optimiser.zero_grad()
net.train()        
_,forecast = net(torch.tensor(feature, dtype=torch.float).to(DEVICE))
loss = F.l1_loss(torch.t(torch.mul(torch.t(forecast), 
                                           torch.tensor(weight,dtype=torch.float).to(DEVICE))) ,
                         torch.t(torch.mul(torch.t(torch.tensor(target,dtype=torch.float).to(DEVICE)), 
                                           torch.tensor(weight,dtype=torch.float).to(DEVICE))),reduction='mean')
loss.backward()
params.append(net.parameters())
optimiser.step()
pytorch python
1 Answers

Yes, it will be correct.

If you are not using in-place operations, the gradients will be computed correctly. Besides, in the current version of Pytorch, there will be an error reported if you accidentally involve some in-place operations into your program.

Here is a related discussion. You can find more information here.

user_66241010
Related