I am using the Matcher class and spaCy v3.1.3. to extract patterns from sentences.
I have a label called "Inteval" and two patterns that I want to capture. However, the largest pattern (the second "pattern" shown below) is made up of a sub-pattern specified in the first "pattern" and I wish that when there are matches from the largest pattern, only extract the largest pattern and not the subpatterns.
For example, I have sentences containing strings of the type: "500 km até 543 km", "13 a 22 km", "550 km - 500 km" and "568 km até 420 km até 190 km". I want all of them to have the same label ("Interval"). In the case of: "568 km até 420 km até 190 km", is there a way to extract just "568 km até 420 km até 190 km" without also extracting: "568 km até 420 km", "420 km até 190 km "?
import spacy
from spacy.matcher import Matcher
nlp = spacy.load('pt_core_news_lg')
UN = ['km']
TEXTS = ['500 km até 543 km', 'de 568 km até 420 km até 190 km', 'de 700 km até 400 km até 100 km']
matcher = Matcher(nlp.vocab)
# Examples:
# 500 km até 543 km
# 13 a 22 km
# 550 km - 500 km
pattern = []
pattern.append([{'POS': 'NUM'},
{'OP': '?', 'TEXT': {'IN': UN}},
{'TEXT': {'IN': ['a', 'até', '-']}},
{'POS': 'NUM'},
{'TEXT': {'IN': UN}}
])
# Example:
# 568 km até 420 km até 190 km
pattern.append([{'POS': 'NUM'},
{'OP': '?', 'TEXT': {'IN': UN}},
{'TEXT': {'IN': ['a', 'até']}},
{'POS': 'NUM'},
{'TEXT': {'IN': UN}},
{'TEXT': {'IN': ['a', 'até']}},
{'POS': 'NUM'},
{'TEXT': {'IN': UN}}
])
matcher.add('Intervalo', pattern)
TRAINING_DATA = []
for doc in nlp.pipe(TEXTS):
spans = [doc[start:end] for match_id, start, end in matcher(doc)]
entities = [(span.start_char, span.end_char, "Intervalo") for span in spans]
training_example = (doc.text, {"entities": entities})
TRAINING_DATA.append(training_example)
print(*TRAINING_DATA, sep="\n")
Results:
('500 km até 543 km', {'entities': [(0, 17, 'Intervalo')]})
('de 568 km até 420 km até 190 km', {'entities': [(3, 20, 'Intervalo'), (3, 31, 'Intervalo'), (14, 31, 'Intervalo')]})
('de 700 km até 400 km até 100 km', {'entities': [(3, 20, 'Intervalo'), (3, 31, 'Intervalo'), (14, 31, 'Intervalo')]})