I have a document of text and I want to find out which person the text is "most about", my approximation of "most about" will be defined as the person mentioned most.
I use Spacy Named Entity Recognition (NER) to get a list of all NER, then filter for type == 'PERSON'. I then get a frequency distribution for each person. This works ok, but Spacy treats each literal named entity token as different, i.e. "John Smith" is not "John" or "Smith".
- Is there a better approach to find the person the text is "most about"?
- How can I get a merged/combined count where different forms of the same names are combined?
- I can come up with ways to do this myself but is there a specific name for this concept, so I can research further?
- Does some NLP library (Spacy or NLTK for example) have the capability to do this?
Here is my current code and output:
import spacy
import nltk
# Load English tokenizer, tagger, parser and NER
nlp = spacy.load("en_core_web_sm")
def get_frequency(word_list):
"""
Get the frequency of each word
"""
freqs = nltk.FreqDist(word_list)
freq_list = [(k, v) for k, v in freqs.items()]
print(freq_list)
print()
def get_ner_frequency(text):
"""
Extract Named Entity References (NER) of type "PERSON" from text
"""
# Parse with spacy
sp_doc = nlp(text)
# Get named people entities
people = [ent.text for ent in sp_doc.ents if ent.label_ == "PERSON"]
return get_frequency(people)
text = """
Mr John Smith is a person.
Smith likes to ride horses.
Mike told John to be careful. J. Smith was very careful.
A man called Smithy told me to get a horse. John Smith fell off one day. John got hurt badly.
"""
get_ner_frequency(text)
Current Output:
[('John Smith', 2), ('Smith', 1), ('Mike', 1), ('John', 2), ('J. Smith', 1), ('Smithy', 1)]
Ideally, I would like output that looks like this:
[('John Smith', 7), ('Mike', 1)]
As with most other NLP problems, I realise that this can not be done perfectly in all cases, I am just after a decent approximation.