do I have to add softmax in def forward when I use torch.nn.CrossEntropyLoss

user_14370815 0
Viewed 575

https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

When I read the contents above, I understood that torch.nn.CrossEntropy already computes exp score of the last layer. So I thought the forward function doesn't have to include softmax. For example, return self.fc(x) rather than return nn.softmax(self.fc(x)). However, I'm confused, for I've seen several implementations of ConvNet Classifiers that use both ways (they return with or without softmax while both use cross entropy loss).

Do this issues affect the performance of Classifier?? Which way is correct?

softmax pytorch python
1 Answers

JHPark,

You are correct - with torch.nn.CrossEntropyLoss there is no need to include softmax layer. If one does include softmax it will still lead to proper classification result, since softmax does not change which element has max score. However, if applied twice, it may distort relative levels of the outputs, making gradients less strong and potentially slowing training a bit.

user_103964690
Related