Merge name forms for same person found via NER with Spacy

Viewed 296

I have a document of text and I want to find out which person the text is "most about", my approximation of "most about" will be defined as the person mentioned most.

I use Spacy Named Entity Recognition (NER) to get a list of all NER, then filter for type == 'PERSON'. I then get a frequency distribution for each person. This works ok, but Spacy treats each literal named entity token as different, i.e. "John Smith" is not "John" or "Smith".

  1. Is there a better approach to find the person the text is "most about"?
  2. How can I get a merged/combined count where different forms of the same names are combined?
  3. I can come up with ways to do this myself but is there a specific name for this concept, so I can research further?
  4. Does some NLP library (Spacy or NLTK for example) have the capability to do this?

Here is my current code and output:

import spacy
import nltk


# Load English tokenizer, tagger, parser and NER
nlp = spacy.load("en_core_web_sm")


def get_frequency(word_list):
    """
    Get the frequency of each word
    """
    freqs = nltk.FreqDist(word_list)
    freq_list = [(k, v) for k, v in freqs.items()]
    print(freq_list)
    print()


def get_ner_frequency(text):
    """
    Extract Named Entity References (NER) of type "PERSON" from text

    """
    # Parse with spacy
    sp_doc = nlp(text)

    # Get named people entities
    people = [ent.text for ent in sp_doc.ents if ent.label_ == "PERSON"]

    return get_frequency(people)


text = """
Mr John Smith is a person.
Smith likes to ride horses.
Mike told John to be careful. J. Smith was very careful.
A man called Smithy told me to get a horse. John Smith fell off one day. John got hurt badly.
"""

get_ner_frequency(text)


Current Output:

[('John Smith', 2), ('Smith', 1), ('Mike', 1), ('John', 2), ('J. Smith', 1), ('Smithy', 1)]

Ideally, I would like output that looks like this:

[('John Smith', 7), ('Mike', 1)]

As with most other NLP problems, I realise that this can not be done perfectly in all cases, I am just after a decent approximation.

1 Answers

As the comment mentioned, what you want to do is called "coreference resolution". spaCy doesn't have a built-in coreference model yet, but you can try coreferee.

Related