How does Keras deal with log(0) for categorical cross entropy?

user_7960084 0
Viewed 1767

I have a neural network, trained on MNIST, with categorical cross entropy as its loss function.

For theoretical purposes my output layer is ReLu. Therefore a lot of its outputs are 0.

Now I stumbled across the following question:

Why don't I get a lot of errors, since certainly there will be a lot of zeros in my output, which I will take the log of.

Here, for convenience, the formula for categorical cross entropy.

L = \sum_{i=1}^m \sum_j L_{i,j} \log y_{i,j}

cross-entropy loss-function keras
2 Answers
user_5627690

Keras clips the network output using a constant 1e-7 and adds this constant again to the clipped output before performing the logarithm operation as defined here.

  epsilon_ = _constant_to_tensor(epsilon(), output.dtype.base_dtype)
  output = clip_ops.clip_by_value(output, epsilon_, 1. - epsilon_)

  # Compute cross entropy from probabilities.
  bce = target * math_ops.log(output + epsilon())
  bce += (1 - target) * math_ops.log(1 - output + epsilon())
  return -bce

Why Keras adds epsilon again to the clipped output is a mystery to me.

user_5114280
Related