Get the Cross Entropy Loss in pytorch as in Keras

user_1298461 0
Viewed 2140

I am struggling to port a classification model form keras to pytorch. Especially the cross entropy loss seems to return totally different numbers.

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])

print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred)))
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).argsort(dim=-1).float(), t.tensor(y_true).argmax(dim=-1)))```

prints:

Keras tf.Tensor([2.3369865], shape=(1,), dtype=float32)

PyTorch tensor(1.4587)

Since I have a custom loss function where cross entropy is a part of it, I would need to get similar if not the same numbers.

pytorch keras tensorflow python
1 Answers

The problem is that they have different implementations.

As pytorch docs says, nn.CrossEntropyLoss combines nn.LogSoftmax() and nn.NLLLoss() in one single class. However, tensorflow docs specifies that keras.backend.categorical_crossentropy do not apply Softmax by default unless you set from_logits is True. For this reason, you should not use keras.backend.categorical_crossentropy without having previously apply softmax unless you use from_logits=True.

If you don't want to apply softmax beforehand you should use:

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])


print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred), from_logits=True))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)

Otherwise, you can apply Softmax manually before computing categorical_crossentropy

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])


print("Keras", K.categorical_crossentropy(K.constant(y_true), K.softmax(K.constant(y_pred))))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)

So you should not use keras.backend.categorical_crossentropy with from_logits=False as you were doing in your example.

tf.keras.backend.categorical_crossentropy

target: A tensor of the same shape as output.

output: A tensor resulting from a softmax (unless from_logits is True, in which case output is expected to be the logits).

from_logits: Boolean, whether output is the result of a softmax, or is a tensor of logits.

user_57142550
Related