Spacy matching priority

user_19433022 0
Viewed 57

I'm looking to create a physics pattern library with spacy: I want to detect time and speed pattern. My aim is to stay flexible with those pattern.

time_pattern = [
    [
    {'LIKE_NUM': True, 'OP': '?'},
    {'LOWER':{'IN': ['time', 's','h','min']}},
    {'LOWER': {"IN": ['maximum','minimum','min','max']}, 'OP':'?'}
    ]
]
speed_pattern = [
    [
    {'LIKE_NUM': True, 'OP': '?'},
    {'LOWER':{"IN": ['km', 'm']}},
    {'IS_PUNCT': True},
    {'LOWER' : {"IN": ['h','hour','s','min']}}
    ]
]


matcher=Matcher(nlp.vocab, validate =True)
matcher.add("SPEED", speed_pattern)
matcher.add("TIME", time_pattern)

doc=nlp("a certain time, more about 23 min, can't get above 25 km/h")

for id_match, start, end in matcher(doc):
    match_label=nlp.vocab[id_match].text
    print(match_label, '<--', doc[start:end])

So far my code returns this collection of matches:

  • TIME <-- time
  • TIME <-- 23 min
  • TIME <-- min
  • SPEED <-- 25 km/h
  • SPEED<-- km/h
  • TIME <-- h

I want the matcher to match only once, and to match "23 min" rather than "min". Also would like the matcher not to match an element already matched ( for exemple "h" should not be matched because it already matched in "km/h"

spacy nlp matcher
2 Answers

You can try add greedy="LONGEST" to matcher.add() to return only the longest (or FIRST) matches:

matcher.add("SPEED", speed_pattern, greedy="LONGEST")
matcher.add("TIME", time_pattern, greedy="LONGEST")

But note that this doesn't handle overlaps across different match IDs:

TIME <-- 23 min
TIME <-- time
TIME <-- h
SPEED <-- 25 km/h

If you want to filter all of the matches, you can use matcher(doc, as_spans=True) to get the matches directly as spans and then use spacy.util.filter_spans to filter the whole list of spans to a list of non-overlapping spans with the longest spans preferred: https://spacy.io/api/top-level#util.filter_spans

[time, 23 min, 25 km/h]
user_4618470

You can use as_spans=True option with spacy.matcher.Matcher (introduced in spaCy v3.0):

matches = matcher(doc, as_spans=True)
for span in spacy.util.filter_spans(matches):
    print(span.label_, "->", span.text)

From the documentation:

Instead of tuples, return a list of Span objects of the matches, with the match_id assigned as the span label. Defaults to False.

See the Python demo:

import spacy
from spacy.tokens.doc import Doc
from spacy.matcher import Matcher

nlp = spacy.load('en_core_web_sm')
time_pattern = [
    [
        {'LIKE_NUM': True, 'OP': '?'},
        {'LOWER':{'IN': ['time', 's','h','min']}},
        {'LOWER': {"IN": ['maximum','minimum','min','max']}, 'OP':'?'}
    ]
]
speed_pattern = [
    [
        {'LIKE_NUM': True, 'OP': '?'},
        {'LOWER':{"IN": ['km', 'm']}},
        {'IS_PUNCT': True},
        {'LOWER' : {"IN": ['h','hour','s','min']}}
    ]
]

matcher=Matcher(nlp.vocab, validate =True)
matcher.add("SPEED", speed_pattern)
matcher.add("TIME", time_pattern)

doc=nlp("a certain time, more about 23 min, can't get above 25 km/h")

matches = matcher(doc, as_spans=True)
for span in spacy.util.filter_spans(matches):
    print(span.label_, "->", span.text)

Output:

TIME -> time
TIME -> 23 min
SPEED -> 25 km/h
user_38329700
Related