Match only the largest pattern and not sub patterns with Matcher spaCy

Viewed 139

I am using the Matcher class and spaCy v3.1.3. to extract patterns from sentences.

I have a label called "Inteval" and two patterns that I want to capture. However, the largest pattern (the second "pattern" shown below) is made up of a sub-pattern specified in the first "pattern" and I wish that when there are matches from the largest pattern, only extract the largest pattern and not the subpatterns.

For example, I have sentences containing strings of the type: "500 km até 543 km", "13 a 22 km", "550 km - 500 km" and "568 km até 420 km até 190 km". I want all of them to have the same label ("Interval"). In the case of: "568 km até 420 km até 190 km", is there a way to extract just "568 km até 420 km até 190 km" without also extracting: "568 km até 420 km", "420 km até 190 km "?

import spacy
from spacy.matcher import Matcher

nlp = spacy.load('pt_core_news_lg')

UN = ['km']

TEXTS = ['500 km até 543 km', 'de 568 km até 420 km até 190 km', 'de 700 km até 400 km até 100 km']

matcher = Matcher(nlp.vocab)

# Examples: 
# 500 km até 543 km
# 13 a 22 km
# 550 km - 500 km
pattern = []
pattern.append([{'POS': 'NUM'}, 
                {'OP': '?', 'TEXT': {'IN': UN}}, 
                {'TEXT': {'IN': ['a', 'até', '-']}}, 
                {'POS': 'NUM'}, 
                {'TEXT': {'IN': UN}}
                ]) 

# Example:
# 568 km até 420 km até 190 km    
pattern.append([{'POS': 'NUM'}, 
                {'OP': '?', 'TEXT': {'IN': UN}}, 
                {'TEXT': {'IN': ['a', 'até']}}, 
                {'POS': 'NUM'}, 
                {'TEXT': {'IN': UN}},
                {'TEXT': {'IN': ['a', 'até']}}, 
                {'POS': 'NUM'}, 
                {'TEXT': {'IN': UN}}                     
                ]) 
    
matcher.add('Intervalo', pattern)

TRAINING_DATA = []

for doc in nlp.pipe(TEXTS):
    spans = [doc[start:end] for match_id, start, end in matcher(doc)]
    entities = [(span.start_char, span.end_char, "Intervalo") for span in spans]
    training_example = (doc.text, {"entities": entities})
    TRAINING_DATA.append(training_example)

print(*TRAINING_DATA, sep="\n")

Results:

('500 km até 543 km', {'entities': [(0, 17, 'Intervalo')]})
('de 568 km até 420 km até 190 km', {'entities': [(3, 20, 'Intervalo'), (3, 31, 'Intervalo'), (14, 31, 'Intervalo')]})
('de 700 km até 400 km até 100 km', {'entities': [(3, 20, 'Intervalo'), (3, 31, 'Intervalo'), (14, 31, 'Intervalo')]})
1 Answers

You're using the Matcher directly, but if you use the EntityRuler instead (which just as a Matcher inside) it will handle this for you automatically - when matches overlap it takes the longest one, or the one that starts earlier if length is equal.

Related