Spacy - create a nested taxonomy using PhraseMatcher

user_3925023 0
Viewed 208

I'm using Spacy PhraseMatcher function to detect some specific taxonomy from a list of short text. This is a sample list:

SK-washer SKM16-FSt-Geomet 321A
SK-washer SKM20-FSt-Geomet 321A
SK-washer SKM24-FSt-Geomet 321A
Hexagon head bolt M12x80 ISO 4014-8.8-Geomet 321A+VL
Hexagon head bolt M12x90 ISO 4014-8.8-Geomet 321A+VL
Hexagon head bolt M20x90 ISO 4014-8.8-Geomet 3

I'm using this code:

import spacy
from spacy.matcher import PhraseMatcher
from spacy.tokens import Span
nlp = spacy.blank('en')

mat_type = [nlp(text) for text in ('screw', 'bolt', 'washer')]
head_type = [nlp(text) for text in ('hexa', 'hexagon')]
geomet_type = [nlp(text) for text in ('321a+vl', '500a', '321a')]
iso_norm = [nlp(text) for text in ('4014','4014-A2-70','4017-8.8', '7040', '7040-8', '7042-10')]

#iso_4014_A2 = [nlp(text) for text in ('ISO', '4014','A2','70')]

matcher = PhraseMatcher(nlp.vocab)
matcher.add('MAT_TYPE', None, *mat_type)
matcher.add('HEAD_TYPE', None, *head_type)
matcher.add('GEOMET_TYPE', None, *geomet_type)
matcher.add('ISO_NORM', None, *iso_norm)
#matcher.add('ISO_4014_A2', None, *iso_4014_A2)
#https://spacy.io/usage/rule-based-matching
with open('./sample_list.txt', 'r') as infile:
    data = infile.readlines()
    for i in data:
        print(i)
        doc = nlp(i.lower())
        matches = matcher(doc)
        for match_id, start, end in matches:
            rule_id = nlp.vocab.strings[match_id]  # get the unicode ID, i.e. 'COLOR'
            span = doc[start : end]  # get the matched slice of the doc
            print(rule_id, span.text)

So I'm able to produce this output:

Hexagon head bolt M12x90 ISO 4014-8.8-Geomet 321A+VL

HEAD_TYPE hexagon
MAT_TYPE bolt
ISO_NORM 4014
GEOMET_TYPE 321a+vl

I need to create a nested taxonomy, so for example I want to recognize as "Hexagon" all terms that falls into same concept entity ("Hexa", "Hex"). I'm unable to figure out how to do it with PhraseMatcher or if it's better to use an alternative approach.

Thanks in advance for any suggestion.

###UPDATE As suggested I modified the code:

ruler = nlp.add_pipe("entity_ruler")
patterns = [{"label": "MAT_TYPE", "pattern": [{"LOWER": "screw"}], "id": "Screw"},
            {"label": "MAT_TYPE", "pattern": [{"LOWER": "bolt"}], "id": "Bolt"},
           {"label": "MAT_TYPE", "pattern": [{"LOWER": "washer"}], "id": "Washer"},
           {"label": "ISO", "pattern": [{"TEXT": "4014"}, {"TEXT": "4014-8.8"}], "id": "4014"},
            {"label": "ISO", "pattern": [{"LOWER": "4017-8.8"}], "id": "4017"},
           {"label": "ISO", "pattern": [{"LOWER": "7040"}, {"LOWER": "7040-8"}], "id": "7040"}]
ruler.add_patterns(patterns)
with open('./sample_list.txt', 'r') as infile:
    data = infile.readlines()
    for i in data:
        print(i)
        doc = nlp(i.lower())        
        print([(ent.text, ent.label_, ent.ent_id_) for ent in doc.ents])

Seems do exactly what I need except I'm unable to detect label that are included in text (ISO):

[('washer', 'MAT_TYPE', 'Washer')]
Hexagon head bolt M12x80 ISO 4014-8.8-Geomet 321A+VL

[('bolt', 'MAT_TYPE', 'Bolt')]
Hexagon head bolt M12x90 ISO 4014-8.8-Geomet 321A+VL

[('bolt', 'MAT_TYPE', 'Bolt')]
Hexagon head bolt M20x90 ISO 4014-8.8-Geomet 321A+VL
spacy python
1 Answers

It sounds like you want to be able to add metadata to matches to identify things that are the same in your scheme. To do that you should have a look at adding IDs to match patterns using the EntityRuler. Here's an example from the spaCy docs:

from spacy.lang.en import English

nlp = English()
ruler = nlp.add_pipe("entity_ruler")
patterns = [{"label": "ORG", "pattern": "Apple", "id": "apple"},
            {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "francisco"}], "id": "san-francisco"},
            {"label": "GPE", "pattern": [{"LOWER": "san"}, {"LOWER": "fran"}], "id": "san-francisco"}]
ruler.add_patterns(patterns)

doc1 = nlp("Apple is opening its first big office in San Francisco.")
print([(ent.text, ent.label_, ent.ent_id_) for ent in doc1.ents])

doc2 = nlp("Apple is opening its first big office in San Fran.")
print([(ent.text, ent.label_, ent.ent_id_) for ent in doc2.ents])

In this case san-francisco is an ID, but you could add IDs like head-hex to identify all the variations of "hexagon" in your patterns.

Note that I would refer to what you're trying to do as normalization, or possibly fine-grained entity labelling, not as creating a taxonomy. I'm not sure what you mean by "nested taxonomy", I guess you mean that your labels are nested? Like you have a "head" and then within "head" you have different categories like "hex" or "phillips".

It might be helpful if you gave more examples of the kinds of output you want.

The issue with your pattern not getting your ISO patterns is probably because something like 4014-8.8 is ending up as multiple tokens. Change your patterns to be like this:

{"label": "ISO", "pattern": "4017-8.8", "id": "4017"}

When you pass a string, spaCy will figure out the tokenization for you.

Since your patterns here are just numbers it doesn't matter, but if you actually need to match on the LOWER attribute, look at the docs on matching on other attributes.

user_3557150
Related