I get different validation loss for almost same type of accuracy

user_12113684 0
Viewed 58

I made two different convolution neural networks for a multi-class classification. And I tested the performance of the two networks using evaluate_generator function in keras. Both models give me comparable accuracies. One gives me 55.9% and the other one gives me 54.8%. However, the model that gives me 55.9% gives a validation loss of 5.37 and the other 1.24.

  1. How can these test losses be so different when the accuracies are similar. If anything I would expect the loss for the model with 55.9% accuracy to be lower but it's not.

  2. Isn't loss the total sum of errors the network is making?

Insights would be appreciated.

keras deep-learning
2 Answers

Isn't loss the total sum of errors the network is making?

Well, not really. Loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some "cost" associated with the event.

For exaple, in regression tasks loss function can be mean squared error. In classification - binary or categorical crossentropy. These loss functions measure how your model understanding of data is close to the reality.

Why both loss and accuracy are high?

High loss doesn't mean you model don't know anything. In basic case you can think about it that the smaller the loss, the more confident the model is in its choice.

So model with the higher loss not really sure about its answers.

You can also read this discussion about high loss and accuracy

user_115891830

Even though the accuracies are similar, the loss value is not correlated when comparing different models.

Accuracy measures the fraction of correctly classified samples over the Total Population of your samples.

With regards to the loss value, from keras documentation:

Return value

For scalars, the loss value of the test (if the model does not have a merit function) or > a list of scalars (if the model computes another merit function).

If this doesn't help on your case (I don't have a way to reproduce the issue), please check the following known issues in keras, with regards to the evaluate_generator function: evaluate_generator

user_135265120
Related