Loss of Multi-Output Model in Pytorch

user_8444465 0
Viewed 2421

I have a multi-output model in PyTorch when I train them using the same loss and then to backpropagate I combine the loss of both the output but when one output loss decreases others increase and so on. How can I fix the problem?

def forward(self, x):
    #neural network arch. forward pass
    x = dense1(x)
    x1 = dense2(x)
    x2 = dense2(x)
    x1 = F.log_softmax(x1) 
    x2 = F.log_softmax(x2)
    return x1, x2

out1, out2 = model(data)
loss1 = NLLL(out1, target1)
loss2 = NLLL(out2, target2)
loss = loss1 + loss2
loss.backward()

When loss1 decrease loss2 increase and when loss2 decrease loss1 increase how can I fix the issue. Can any other operator other than '+' be used to combine the loss or should I put weights to the different loss?

pytorch deep-learning machine-learning
1 Answers

If you have two different loss functions, and you finish the forwards for both of them separately, it is smart to do 

(loss1 + loss2).backward()

This is computationally efficient.

What you should achieve is to make your model learn, how to minimize the loss. Some code from your example is absent, but you should have the nn.Module, probable your custom module with parameters inside that should learn to lower to loss.

I liked your approach summing the loss = loss1 + loss2.

user_58849550
Related