Why does binary_crossentropy loss not match between fitting/evaluation and manual check?

user_2364295 0
Viewed 157

When I train or evaluate with binary_crossentropy I'm getting huge values, but when I put the results through manually it looks normal. What am I doing wrong? I think binary_crossentropy can't exceed about 15.5 in any case.

test_in, test_out1 = makeTestData()
test_out2 = np.zeros((1000,1))
model.compile(..., loss=['mean_squared_error', 'binary_crossentropy'], loss_weights=[1.0, 0.001])

model.evaluate(test_in,[test_out1, test_out2])
# Result: [0.3238, 0.106, 217.815]

pred = model.predict(test_in)
# np.max(pred[1]) = 1.0, np.min(pred[1]) = 0.0019
K.mean(tf.losses.binary_crossentropy(test_out2, pred[1]))
# Result: 13.31

217.8 loss in evaluation (and training, not shown), but only 13.3 when run manually through.

keras tensorflow
1 Answers

It's difficult to determine what might be the issue here with such a minimal example. I have 2 theories:

  1. Your data is not the same when running manually vs not manually. Something is amiss here, and it's blowing up because of that.

  2. Your evaluation mode is different when running 'manually', vs not manually, and you've fallen into this very particular edgecase:

  if (not isinstance(output, (ops.EagerTensor, variables_module.Variable)) and
      output.op.type == 'Sigmoid') and not hasattr(output, '_keras_history'):
    # When sigmoid activation function is used for output operation, we
    # use logits from the sigmoid function directly to compute loss in order
    # to prevent collapsing zero when training.
    # ... computes using logits

  # ... computes manually with clipping

If the latter looks to be true, try passing the logits directly without doing a sigmoid to see if you can end up with consistent values. In addition, you might try passing in the from_logits=True flag

user_13690190
Related