I am trying to compute inter-annotator agreement on a toy example using NLTK's nltk.metrics.agreement module.
Specifically I am trying to compute agreement using the alpha metric (Krippendorff) using two different distance metrics(binary_distance and interval_distance).
The expected result of toy example 1 below, which has near total agreement (only one pair disagrees), is a value close to 1. However, in both cases res is 0.0. Why?
I understand that Krippendorff's alpha is designed for intervals rather than binary-like two-category labels. I would however not expect a zero agreement value back from the module. For the background, the toy example is simply a specific subset of a larger dataset containing annotation scores in the range [1,4]. The subset belongs to a particular population within that dataset.
In toy example 2 things start to look better for the interval alpha. Binary alpha should probably raise an exception given that there are now three labels in the data.
Toy Example 1
from nltk.metrics.agreement import AnnotationTask
from nltk.metrics import interval_distance, binary_distance
annotation_triples = [('coder_1', '1', 4),
('coder_2', '1', 4),
('coder_1', '2', 4),
('coder_2', '2', 4),
('coder_1', '3', 4),
('coder_2', '3', 4),
('coder_1', '4', 4),
('coder_2', '4', 3)]
t = AnnotationTask(annotation_triples, distance=binary_distance)
result = t.alpha()
t = AnnotationTask(annotation_triples, distance=interval_distance)
result = t.alpha()
result binary: 0.0
result interval: 0.0
Toy Example 2 (replaced first pair using 1 instead of 4)
annotation_triples = [('coder_1', '1', 1),
('coder_2', '1', 1),
('coder_1', '2', 4),
('coder_2', '2', 4),
('coder_1', '3', 4),
('coder_2', '3', 4),
('coder_1', '4', 4),
('coder_2', '4', 3)]
result binary: 0.59
result interval: 0.93