How to generate Precision, Recall and F-score in Named Entity Recognition using Spacy v3? Seeking ents_p, ents_r, ents_f for a small custom NER model

user_7084804 0
Viewed 707

The example code is given below, you may add one or more entities in this example for training purposes (You may also use a blank model with small examples for demonstration). I am seeking a complete working solution for custom NER model evaluation (precision, recall, f-score), Thanks in advance to all NLP experts.

import spacy
nlp = spacy.load('en_core_web_sm')
example_text = "Agra is famous for Tajmahal, The CEO of Facebook will visit India shortly to meet Murari Mahaseth and to visit Tajmahal"
ner = nlp.get_pipe("ner")
doc = nlp(example_text)
for e in doc.ents:
    print(e.text + ' - ' + str(e.start_char) + ' - ' + str(e.end_char) + ' - ' + e.label_ + ' - ' + str(
        spacy.explain(e.label_)))
precision-recall spacy-3 named-entity-recognition python
1 Answers

I will give a brief example :

import spacy
from spacy.scorer import Scorer
from spacy.tokens import Doc
from spacy.training.example import Example

examples = [
    ('Who is Talha Tayyab?',
     {(7, 19, 'PERSON')}),
    ('I like London and Berlin.',
     {(7, 13, 'LOC'), (18, 24, 'LOC')}),
     ('Agra is famous for Tajmahal, The CEO of Facebook will visit India shortly to meet Murari Mahaseth and to visit Tajmahal.',
     {(0, 4, 'LOC'), (40, 48, 'ORG'), (60, 65, 'GPE'), (82, 97, 'PERSON'), (111, 119, 'GPE')})
]

def my_evaluate(ner_model, examples):
    scorer = Scorer()
    example = []
    for input_, annotations in examples:
        pred = ner_model(input_)
        print(pred,annotations)
        temp = Example.from_dict(pred, dict.fromkeys(annotations))
        example.append(temp)
    scores = scorer.score(example)
    return scores

ner_model = spacy.load('en_core_web_sm') # for spaCy's pretrained use 'en_core_web_sm'
results = my_evaluate(ner_model, examples)
print(results)

As, I said this is just an example you can make changes according to your needs.

user_130861280
Related