How to calculate the recall and precision in Entity Linking?

user_12242326 0
Viewed 186

I am pretty confused about how to calculate the recall and precision in the process of entity linking. When we disambiguate an entity to the correct sense(T) or incorrect sense(F), we only have this two situation. But how can we get those four situations: TP, TN, FP, FN ?

nlp
1 Answers

This is an interesting question. I think the reason why the answer was avoiding you lies in the fact there are more than just two correct/not answers for the evaluation of your prediction.

Entity linking is a form of multiclass classification where there could be many possible answers, but only one of them is correct. Following the example in wikipedia, "I don't like Paris" could mean "Paris, France", "Paris, Texas" or "Paris Hilton".

Evaluating the performance of a multiclass classifier follows on from how you score a binary classifier (which only has two possible outcomes):

Table 1

             Actual
              0   1
Predicted 0  TN  FN
          1  FP  TP

Once you have this you can track the rates, in the TPR, FPR etc. for recall, precision etc. which you sound familiar with.

To evaluate a multiclass classifier, you need to evaluate it per possible answer (class) and treat all incorrect answers as negatives. (ref: Microsoft, SciKitLearn )

Imagine A, B and C are the three above Paris entities (for readability). I will add the "binary treatment" next to the classification, so you can see the score is following the binary, in the same way as above.

First you would score the linker for A: Paris France.

Table 2A

               Actual
               A   B   C
               1   0   0
Predicted A 1  TP  FP  FP
          B 0  FN  TN  TN
          C 0  FN  TN  TN

Next score the B answer:

Table 2B

              Actual
              A   B   C
              0   1   0
Predicted A 0 TN  FN  TN
          B 1 FP  TP  FP
          C 0 TN  FN  TN

Finally considering C

Table 2C

              Actual
              A   B   C
              0   0   1
Predicted A 0 TN  TN  FN
          B 0 TN  TN  FN
          C 1 FP  FP  TP

Of course in your evaluator, you would automate this treatment. Good frameworks have the evaluation tools built in which will do this under the hood.

(By the way, I just imagined and created those tables as a way of illustrating it, so apologies for any errata).

user_2092880
Related