Spacy Rule Based Matching Issue

user_8688214 0
Viewed 160

I am trying to extract a phrase from textual data. I am currently using SpaCy Rule-Based Matching. This is working fine until I saw that "Hiv-1 dna quant", this phrase is not getting detected. The patter I used is shown in the below code.

matcher = Matcher(nlp.vocab)
pattern = [{'LOWER': 'hiv'}, {"IS_PUNCT": True}, {"TEXT": {"REGEX":"\d{1,2}"}},
         {'LOWER': 'dna'},
         {'LOWER': 'quant'}]
matcher.add("HelloWorld", [pattern])
data = "probe Hiv-1 dna amp probe Hiv-1 dna quant Hiv-2 dna dir probe Hiv-2 dna"
doc = nlp(data)
matches = matcher(doc)
for match_id, start, end in matches:
    string_id = nlp.vocab.strings[match_id]  # Get string representation
    span = doc[start:end]  # The matched span
    print(span.text)

I have also tried the following pattern

pattern = [{"LOWER": "hiv"}, {"IS_PUNCT": True}, {"LOWER":"1"}, {"LOWER": "dna"}, {"LOWER":"quant"}]

But it doesnt detect it.

Is there any other way to do this?

spacy python
1 Answers

When you have such issues, first make sure you understand how Spacy tokenizes your string. Look:

>>> [t for t in doc]
[probe, Hiv-1, dna, amp, probe, Hiv-1, dna, quant, Hiv-2, dna, dir, probe, Hiv-2, dna]

So, your Hiv-1 is a single token. Now, you need to add another pattern that would account for the fact that {'LOWER': 'hiv'}, {"IS_PUNCT": True}, {"TEXT": {"REGEX":"\d{1,2}"}} can be a single token. For example, it can look like {'LOWER': {"REGEX":"^hiv[\W_]\d{1,2}$"}}, where the lowercased token text must match a ^hiv[\W_]\d{1,2}$ regex.

You can use

patterns = [
    [{'LOWER': 'hiv'}, {"IS_PUNCT": True}, {"TEXT": {"REGEX":"\d{1,2}"}}, {'LOWER': 'dna'}, {'LOWER': 'quant'}],
    [{'LOWER': {"REGEX":"^hiv[\W_]\d{1,2}$"}}, {'LOWER': 'dna'}, {'LOWER': 'quant'}]
]
matcher = Matcher(nlp.vocab)
matcher.add("HelloWorld", patterns)
doc = nlp(data)
print([doc[start:end].text for _, start,end in matcher(doc)])
# => ['Hiv-1 dna quant']

The ^hiv[\W_]\d{1,2}$ regex means

  • ^ - start of the string (here, token)
  • hiv - hiv text
  • [\W_] - any non-alphanumeric char
  • \d{1,2} - one or two digits
  • $ - end of the string (here, token).

See the regex demo.

user_38329700
Related