Suppose you have a neural network with 2 layers A and B. A gets the network input. A and B are consecutive (A's output is fed into B as input). Both A and B output predictions (prediction1 and prediction2) Picture of the described architecture You calculate a loss (loss1) directly after the first layer (A) with a target (target1). You also calculate a loss after the second layer (loss2) with its own target (target2).

Does it make sense to use the sum of loss1 and loss2 as the error function and back propagate this loss through the entire network? If so, why is it "allowed" to back propagate loss1 through B even though it has nothing to do with it?

This question is related to this question https://datascience.stackexchange.com/questions/37022/intuition-importance-of-intermediate-supervision-in-deep-learning but it does not answer my question sufficiently. In my case, A and B are unrelated modules. In the aforementioned question, A and B would be identical. The targets would be the same, too.

(Additional information) The reason why I'm asking is that I'm trying to understand LCNN (https://github.com/zhou13/lcnn) from this paper. LCNN is made up of an Hourglass backbone, which then gets fed into MultiTask Learner (creates loss1), which in turn gets fed into a LineVectorizer Module (loss2). Both loss1 and loss2 are then summed up here and then back propagated through the entire network here.

Even though I've visited several deep learning lectures, I didn't know this was "allowed" or makes sense to do. I would have expected to use two loss.backward(), one for each loss. Or is the pytorch computational graph doing something magical here? LCNN converges and outperforms other neural networks which try to solve the same task.