Encode classifier performance in dataframe and account for NaN

user_16719690 0
Viewed 25

I have a pandas dataframe with the performance of a classifier. In one column (c1) there is the ground truth for that data point, whether the case is positive (A) or negative (B). In another column (c2) there is the classifier's response, which is either positive (a) or negative (b). I want to calculate a column that encodes the performance of the classifier according to the typical four cells of a confusion matrix (true positive, true negative, false positive, false negative).

What is the most efficient way to do this in pandas? Currently I have the following:

import pandas as pd
import numpy as np

data = pd.DataFrame({
    'c1': ['A', 'B', 'A', 'B', 'A', 'B', 'B', 'A', 'B', 'A'],
    'c2': ['a', 'b', np.nan, 'a', 'b', 'a', 'b', 'a', np.nan, 'a']
})

data
        c1  c2
    0   A   a
    1   B   b
    2   A   NaN
    3   B   a
    4   A   b
    5   B   a
    6   B   b
    7   A   a
    8   B   NaN
    9   A   a

data.loc[(data['c1']=='A') & (data['c2']=='a'), 'class'] = 'truepos'
data.loc[(data['c1']=='A') & (data['c2']=='b'), 'class'] = 'falseneg'
data.loc[(data['c1']=='B') & (data['c2']=='a'), 'class'] = 'falsepos'
data.loc[(data['c1']=='B') & (data['c2']=='b'), 'class'] = 'trueneg'
data.loc[data['c2'].isnull(), 'class'] = np.nan
data
    c1  c2  class
0   A   a   truepos
1   B   b   trueneg
2   A   NaN NaN
3   B   a   falsepos
4   A   b   falseneg
5   B   a   falsepos
6   B   b   trueneg
7   A   a   truepos
8   B   NaN NaN
9   A   a   truepos

This works but I don't think it's very good because it's quite repetitive. Is there a better way? e.g. maybe some sort of lambda function?

pandas python
1 Answers

Here is another way to do it with a dict:

matrix = {
    ("A", "a"): "truepos",
    ("A", "b"): "falseneg",
    ("B", "a"): "falsepos",
    ("B", "b"): "trueneg",
}

data["class"] = data.apply(lambda x: matrix.get((x[0], x[1]), np.nan), axis=1)

print(data)
# Output
  c1   c2     class
0  A    a   truepos
1  B    b   trueneg
2  A  NaN       NaN
3  B    a  falsepos
4  A    b  falseneg
5  B    a  falsepos
6  B    b   trueneg
7  A    a   truepos
8  B  NaN       NaN
9  A    a   truepos
user_112460560
Related