Confusion matrix across multiple raters

Question

test_data <- data.frame(event= c("event1","event2","event3","event4","event5","event6","event7"),
                    rater1_1 = c("red", "orange", "red", "purple", "orange", "red", "yellow"),
                    rater2_1 = c("red", "orange", "orange", "purple", "orange", "red", "purple"),
                    rater3_1 = c("red", "red", "yellow", "purple", "orange", "red", "yellow"),
                    rater4_1 = c("orange", "orange", "blue", "orange", "orange", "red", "purple"), 
                    rater5_1 = c("blue", "blue", "purple", "orange", "orange", "blue", "yellow")
                    )

with the above data, i'm trying to create a confusion matrix where i can observe disagreement across all the raters for each event. that is, for event1, 3 raters gave "red" and 1 "orange" and 1 "blue."

i believe the best way to approach this would be to take each rater pair comparison (rater1 on y-axis and rater2 on x-axis) and then iterate and tally across all rater pairs.

i'm hoping for something that looks like below:

        red  orange  blue  yellow  purple
red      22    6      2      3      2
orange   6     13     1      4      1
blue     2     1      10     3      1
yellow   3     4      3      9      2
purple   2     1      1      2      9

(note: these values are made up, i didn't manually count the above)

i'm not even sure where to start. most confusion matrices i searched are comparing actual model output to predicted model output (e.g., link). any suggestions would be greatly appreciated.

user_3301113 · Answer

For this solution I am using the dplyr and purrr packages

library(dplyr)
library(purrr)
# convert to long format
df_long <- test_data %>% pivot_longer(-event)

# df_long
# # A tibble: 35 x 3
#   event  name     value 
#          
# 1 event1 rater1_1 red   
# 2 event1 rater2_1 red   
# 3 event1 rater3_1 red   
# 4 event1 rater4_1 orange
# 5 event1 rater5_1 blue  
# 6 event2 rater1_1 orange
# 7 event2 rater2_1 orange
# 8 event2 rater3_1 red   
# 9 event2 rater4_1 orange
#10 event2 rater5_1 blue  
# # ... with 25 more rows

# create function to compute the confusion matrix for two given events
create_confusion_matrix <- function(raters){
 df_long %>% filter(name %in% raters) %>% 
             pivot_wider(names_from=name,values_from=value) %>% 
             select(-event) %>% 
             table()
}

# lets try this function with rater1_1 and rater2_1
create_confusion_matrix(c('rater1_1','rater2_1'))
#        rater2_1
#rater1_1 orange purple red yellow blue
#  orange      2      0   0      0    0
#  purple      0      1   0      0    0
#  red         1      0   2      0    0
#  yellow      0      1   0      0    0
#  blue        0      0   0      0    0


# now we need to get all combinations of two raters
raters2 <- combn(unique(df_long$name),2,simplify=FALSE)


# raters2 is a list, each element is a vector containing 2 raters

# loop over the list and apply create_confusion_matrix for each element
result_list <- map(raters2,create_confusion_matrix)
# result_list is a list, each element is a confusion matrix

#we can them sum all theses tables

contingency <- Reduce('+',result_list)
#        rater2_1
#rater1_1 orange purple red yellow blue
#  orange     14      1   2      1    5
#  purple      6      4   0      3    0
#  red         5      1   9      1    9
#  yellow      0      4   0      3    1
#  blue        0      1   0      0    0

# getting rid of rater1_1 and rater2_1 in dimnames
dimnames(contingency) <- list(dimnames(contingency)[[1]],dimnames(contingency)[[2]])
#       orange purple red yellow blue
#orange     14      1   2      1    5
#purple      6      4   0      3    0
#red         5      1   9      1    9
#yellow      0      4   0      3    1
#blue        0      1   0      0    0

# sum symmetric cells and make contingency table lower triangular
# first lets extract the diagonal
# diag is needed twice, first to extract the diagonal from contingency as a vector
# second to convert this vector to a diagonal matrix
diag_contingency <- diag(diag(contingency))
# sum lower and upper matrices by adding the transposed matrix
# and substracting the diagonal (otherwise added twice)
contingency <- contingency + t(contingency) - diag_contingency
# we know have a symmetrical matrix
#        orange purple red yellow blue
#orange     14      7   7      1    5
#purple      7      4   1      7    1
#red         7      1   9      1    9
#yellow      1      7   1      3    1
#blue        5      1   9      1    0

# set the upper triangular matrix to 0
contingency[upper.tri(contingency)] <- 0

# we get this matrix in the end
contingency
#           orange purple red yellow blue
#orange     14      0   0      0    0
#purple      7      4   0      0    0
#red         7      1   9      0    0
#yellow      1      7   1      3    0
#blue        5      1   9      1    0

Confusion matrix across multiple raters

1 Answers