According to scikit learn documentation, the cohen kappa score can be calculated as this:
from sklearn.metrics import cohen_kappa_score
y_true = [1, 0, 1, 1, 1, 1]
y_pred = [1, 0, 1, 1, 1, 1]
print(cohen_kappa_score(y_true, y_pred)
1
where the 0 and 1 are the labels assigned by annotators.
however, if both annotators never assign the second label(e.g. 0) then the score is nan ! which should be 1 (if i am not mistaken) since both annotators agreed.
from sklearn.metrics import cohen_kappa_score
y_true = [1, 1, 1, 1, 1, 1]
y_pred = [1, 1, 1, 1, 1, 1]
print(cohen_kappa_score(y_true, y_pred)
packages/sklearn/metrics/_classification.py:604: RuntimeWarning: invalid value encountered in true_divide
k = np.sum(w_mat * confusion) / np.sum(w_mat * expected)
nan
What am i missing?
Update: I found this in the github issues: https://github.com/scikit-learn/scikit-learn/issues/9624
Still did not get it that this is a perfect agreement at random, where it should be considered as correct annotation