How to calculate accuracy score of a random classifier?

Viewed 6412

Say for example, a dataset contains 60% instances for "Yes" class and 30% instances for "NO" class. In this scenario, Precision, Recall for the random classifier are

Precision =60%

Recall =50%

Then, what will be the accuracy for random classifier in this scenario?

2 Answers

Some caution is required here, since the very definition of a random classifier is somewhat ambiguous; this is best illustrated in cases of imbalanced data.

By definition, the accuracy of a binary classifier is

acc = P(class=0) * P(prediction=0) + P(class=1) * P(prediction=1)

where P stands for probability.

Indeed, if we stick to the intuitive definition of a random binary classifier as giving

P(prediction=0) = P(prediction=1) = 0.5

then the accuracy computed by the above formula is always 0.5, irrespectively of the class distribution (i.e. the values of P(class=0) and P(class=1)).

However, in this definition, there is an implicit assumption, i.e. that our classes are balanced, each one consisting of 50% of our dataset.

This assumption (and the corresponding intuition) breaks down in cases of class imbalance: if we have a dataset where, say, 90% of samples are of class 0 (i.e. P(class=0)=0.9), then it doesn't make much sense to use the above definition of a random binary classifier; instead, we should use the percentages of the class distributions themselves as the probabilities of our random classifier, i.e.:

P(prediction=0) = P(class=0) = 0.9
P(prediction=1) = P(class=1) = 0.1

Now, plugging these values to the formula defining the accuracy, we get:

acc = P(class=0) * P(prediction=0) + P(class=1) * P(prediction=1)
    = (0.9 * 0.9) + (0.1 * 0.1)
    = 0.82

which is nowhere close to the naive value of 0.5...

As I already said, AFAIK there are no clear-cut definitions of a random classifier in the literature. Sometimes the "naive" random classifier (always flip a fair coin) is referred to as a "random guess" classifier, while what I have described is referred to as a "weighted guess" one, but still this is far from being accepted as a standard...

The bottom line here is the following: since the main reason for using a random classifier is as a baseline, it makes sense to do so only in relatively balanced datasets. In your case of a 60-40 balance, the result turns out to be 0.52, which is admittedly not far from the naive one of 0.5; but for highly imbalanced datasets (e.g. 90-10), the usefulness itself of the random classifier as a baseline ceases to exist, since the correct baseline has become "always predict the majority class", which here would give an accuracy of 90%, in contrast to the random classifier accuracy of just 82% (let alone the 50% accuracy of the naive approach)...

As @desertnaut mentioned, if you're after a naïve benchmark for your model you're always better using "always predict the majority class" as your benchmark, achieving accuracy of %of_samples_in_majority_class (which is always better than either a random guess or a weighted guess).

In Deepchecks (a package I maintain) we have a check that automatically compares the performance of your model to a simple model (either weighted random, majority class or simple decision tree).

from deepchecks.checks import SimpleModelComparison
from deepchecks import Dataset
SimpleModelComparison().run(Dataset(train_df, label='target'), Dataset(test_df, label='target'), model)
Related