I want to calculate the Cohen Kappa score between the two following lists:
[None, 'Anticipation 97 105', None, None, None, 'Disgust 80 87', None, 'Disgust 27 31;32 36', None, 'Disgust 87 93', None, None, None, 'Fear 55 66', 'Joy 158 164', 'Anger 181 185;186 197', None, None, 'Anger 246 249;262 266;267 270', None, None, None, None, 'Sadness 43 57', None, None, 'Disgust 51 52', None, 'Fear 63 69;70 78', 'Joy 83 89', 'Sadness 114 122;123 130', None, None, 'Trust 69 79', 'Fear 127 137;152 156', None, 'Joy 13 19;20 28', 'Anger 11 14;3 7', 'Sadness 96 100', None, 'Sadness 36 45', None, None, 'Sadness 136 142;149 152;153 157', 'Anger 53 58', None, None, None, None, None]
and
['Fear 36 43', 'Surprise 97 105', None, 'Trust 157 168', 'Disgust 0 7', 'Disgust 204 221', None, 'Disgust 27 41', None, 'Disgust 87 93', 'Surprise 13 31', None, 'Anger 104 112;113 123', 'Fear 55 66', 'Joy 53 58;59 63', 'Fear 156 173', 'Sadness 20 39', 'Caring 35 43', 'Sadness 243 270', 'Sadness 65 79', 'Caring 67 91', 'Caring 13 25', 'Caring 14 34', 'Sadness 177 202', 'Disgust 6 19', 'Disgust 180 233', 'Sadness 5 23', 'Sadness 15 21', 'Fear 63 78', 'Caring 10 35', 'Other 0 27', 'Disgust 0 32', 'Anger 66 103', None, 'Sadness 41 48', None, 'Joy 13 45', 'Sadness 11 14', 'Disgust 96 111', None, 'Anger 132 154', 'Anger 53 63', 'Caring 66 87', 'Sadness 85 104', 'Anger 41 94', 'Surprise 68 91', 'Surprise 0 15', 'Caring 0 33', 'Anticipation 0 3', 'Caring 0 22']
These are annotations from several files, None means that the file is empty. I compared each item with the corresponding item from the other list to generate a list of integers, this is the code I used to do this:
anno_list_1 = []
anno_list_2 = []
for i in range(50):
item1 = annotator1[i]
item2 = annotator2[i]
similarity = fuzz.partial_ratio(item1, item2)
if item1 == item2:
anno_list_1.append(1)
anno_list_2.append(1)
elif similarity >= 60:
anno_list_1.append(1)
anno_list_2.append(1)
else:
anno_list_1.append(0)
anno_list_2.append(1)
This gives me the following two lists:
[0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0]
and
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
I am not sure whether this was the best method, as the second list only contains 1's.
I used these lists as the input for Cohen's Kappa score, which gave me a score of 0
cohen_kappa_score(anno_list_1, anno_list_2)
Is there a better way of calculating the Cohen Kappa score for textual annotations or are these annotations just random?