How can I make spaCy matches case Insensitive

user_12496869 0
Viewed 788

How can I make spaCy case insensitive?

Is there any code snippet that i should add or something because I couldn't get entities that are not in uppercase?

import spacy
import pandas as pd

from spacy.pipeline import EntityRuler
nlp = spacy.load('en_core_web_sm', disable = ['ner'])
ruler = nlp.add_pipe("entity_ruler")


flowers = ["rose", "tulip", "african daisy"]
for f in flowers:
    ruler.add_patterns([{"label": "flower", "pattern": f}])
animals = ["cat", "dog", "artic fox"]
for a in animals:
    ruler.add_patterns([{"label": "animal", "pattern": a}])



result={}
doc = nlp("CAT and Artic fox, plant african daisy")
for ent in doc.ents:
        result[ent.label_]=ent.text
df = pd.DataFrame([result])
print(df)
pandas spacy nlp python
2 Answers

As long as it's okay if LOWER is used for all patterns, you can continue to use phrase patterns and add the phrase_matcher_attr option for the entity ruler. Then you don't have worry about tokenizing the phrases and if you have a lot of patterns to match, it will also be faster than using token patterns:

import spacy

nlp = spacy.load('en_core_web_sm', disable=['ner'])
ruler = nlp.add_pipe("entity_ruler", config={"phrase_matcher_attr": "LOWER"})

flowers = ["rose", "tulip", "african daisy"]
for f in flowers:
    ruler.add_patterns([{"label": "flower", "pattern": f}])
animals = ["cat", "dog", "artic fox"]
for a in animals:
    ruler.add_patterns([{"label": "animal", "pattern": a}])

doc = nlp("CAT and Artic fox, plant african daisy")
for ent in doc.ents:
    print(ent, ent.label_)

Output:

CAT animal
Artic fox animal
african daisy flower
user_4618470

You need to create the patterns with LOWER. However, you also need to account for multiword entities, so you need to split your phrases and build the patterns dynamically:

import spacy
import pandas as pd

from spacy.pipeline import EntityRuler
nlp = spacy.load('en_core_web_sm', disable = ['ner'])
ruler = nlp.add_pipe("entity_ruler")

patterns = []
flowers = ["rose", "tulip", "african daisy"]
for f in flowers:
    patterns.append({"label": "FLOWER", "pattern": [{'LOWER': w} for w in f.split()]})
animals = ["cat", "dog", "artic fox"]
for a in animals:
    patterns.append({"label": "ANIMAL", "pattern": [{'LOWER': w} for w in a.split()]})

ruler.add_patterns(patterns)

result={}
doc = nlp("CAT and Artic fox, plant african daisy")
for ent in doc.ents:
        result[ent.label_]=ent.text

print([(ent.text, ent.label_) for ent in doc.ents])

Output:

[('CAT', 'ANIMAL'), ('Artic fox', 'ANIMAL'), ('african daisy', 'FLOWER')]
user_38329700
Related