Inter-rater agreement in Python (Cohen's Kappa)

user_1406413 0
Viewed 28624

I have ratings for 60 cases by 3 raters. These are in lists organized by document - the first element refers to the rating of the first document, the second of the second document, and so on:

rater1 = [-8,-7,8,6,2,-5,...]
rater2 = [-3,-5,3,3,2,-2,...]
rater3 = [-4,-2,1,0,0,-2,...]

Is there a python implementation of Cohen's Kappa somewhere? I couldn't find anything in numpy or scipy, and nothing here on stackoverflow, but maybe I missed it? This is quite a common statistic, so I'm surprised I can't find it for a language like Python.

rating statistics python
6 Answers

To expand on Franck Dernoncourt answer and address skjerns comment here is the code to create a matrix for more than two raters:

import itertools

from sklearn.metrics import cohen_kappa_score
import numpy as np

# Note that I updated the numbers so all Cohen kappa scores are different.
rater1 = [-8, -7, 8, 6, 2, -5]
rater2 = [-3, -5, 3, 3, 2, -2]
rater3 = [-4, -2, 1, 3, 0, -2]

raters = [rater1, rater2, rater3]

data = np.zeros((len(raters), len(raters)))
# Calculate cohen_kappa_score for every combination of raters
# Combinations are only calculated j -> k, but not k -> j, which are equal
# So not all places in the matrix are filled.
for j, k in list(itertools.combinations(range(len(raters)), r=2)):
    data[j, k] = cohen_kappa_score(raters[j], raters[k])

# [[0.        , 0.11764706, 0.        ],
#  [0.        , 0.        , 0.25      ],
#  [0.        , 0.        , 0.        ]]

Here is a plot of data:

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(
    data, 
    mask=np.tri(len(raters)),
    annot=True, linewidths=5,
    vmin=0, vmax=1,
    xticklabels=[f"Rater {k + 1}" for k in range(len(raters))],
    yticklabels=[f"Rater {k + 1}" for k in range(len(raters))],
)
plt.show()

heatmap

user_85003570
Related