According to the PyTorch documentation, the advantage of the class BCEWithLogitsLoss() is that one can use the
log-sum-exp trick for numerical stability.
If we use the class BCEWithLogitsLoss() with the parameter reduction set to None, they have a formula for that:
I now simplified the terms, and obtain after some lines of calculation:
I was curious to see whether this is the way how the Source code does it, but I couldn't find it.. The only code they have is this:
nn.BCEWithLogitsLoss is actually just cross entropy loss that comes inside a sigmoid function. It may be used in case your model's output layer is not wrapped with sigmoid. Typically used with the raw output of a single output layer neuron.
Simply put, your model's output say pred will be a raw value. In order to get probability, you will have to use torch.sigmoid(pred). (To get actual class labels, you need torch.round(torch.sigmoid(pred)).) However, you don't need to do anything like that (i.e take sigmoid) when you use nn.BCEWithLogitsLoss. Here you just have to do the following-
criterion = nn.BCEWithLogitsLoss()
loss = criterion(pred, target) # pred is just raw nn output
Hence, coming to implementation part, criterion accepts two torch tensors - one being the raw nn outputs, the other being the true class labels, then wraps the first using sigmoid - for each element in the tensor and then calculates Cross Entropy loss (-(target*log(sigmoid(pred))) for each pair and reduces it to mean.
All the pytorch functional code is implemented in C++. The source code for the implementation is located here.
The pytorch implementation computes BCEWithLogitsLoss as
where t_n is simply -relu(x). The use of t_n here is basically a clever way to avoid taking exponentials of positive values (thus avoiding overflow). This can be made more clear by substituting t_n into the l_n which yields the following equivalent expression
