Performance Evaluation in Multilabel Classification

user_807231 0
Viewed 603

So I was taking a look at both Accuracy and F1-Measure in order to evaluate a multilabel classification algorithm (each instance is associated with multiple labels). Having:

Accuracy = #Intersection(suggestions,correct_labels) / #Union(suggestions,correct_labels)
F1 Measure = 2 * (P * R) / (P + R)

Which approach is better and why? Under what circumstances should I prefer one to the other?

classification machine-learning performance
1 Answers

Usually, accuracy is easier to interpret. But F1 is more informative.

It depends a lot on the kind of problem you are solving and the relative importance of the different classes.

Eg. In the case of MNIST digit classification where you might expect that the classes are balanced (equal number of examples from each class), the accuracy metric is quite representative of the performance.

But in another case like a test for cancer, you may find that

  1. it is better to find all cases of cancer even if it requires more people who do not have cancer to take the test (high recall but expensive for patients)
  2. there is a lot of imbalance in classes (say 5% of people have cancer) and so you can get a high accuracy by just telling that no one has cancer but this is not helpful at all

In this case, it makes more sense to rely on F1 score and trying to balance precision and recall than looking at just the accuracy.

So it is really depends on the relative importance of each label and what you want to prioritize. Nothing can be said without looking at the actual domain of your problem.

user_66490500
Related