I understand the derivation of partial derivatives for binary cross-entropy.
binary cross entropy
Loss = y * log10(yHat) + (1 - y) * log10(1 - yHat)
dLoss/dyHat = -y/yHat + (1 - y) / (1 - yHat)
categorical cross entropy
Loss = y * log10(yHat)
dLoss/dyHat = -y / (yHat * exp(10))
Though, I do not see the latter derivative used in backpropagation. The problem I am having with it is that only the positive category is updated.
For example; If after the first forward propagation,
y = category 4, and category 3 = [[0, 0, 0, 1], [0, 0, 1, 0]]
yHat = [[0.1, 0.2, 0.1, 0.4], [0.1, 0.3, 0.1, 0.5]]
Then the back propagation, using categorical cross entropy, would be =
[[0., 0., 0., -0.1e-3], [0., 0., -0.4e-3, 0.]
Updating the weights with these values only results in higher numbers at each update, without any penalty for the probabilities of the wrong classifications. Eventually, all of the yHat values approach 1.0.
The same backpropagation step using binary cross entropy gives values =
[[1.1, 1.3, 1.1, -2.5],[1.1, 1.4, -10.0, 2.0]]
Allowing both a reward for the correct category and a penalty for the incorrect.
So, is the practise when using categorical cross entropy to use the binary cross entropy derivative? Doesn’t seem like such a liberty should be taken, but not sure what to do?