pattern match issue with SpaCy

Question

I have the following script using a pattern to match the postcode. However, depending on how I supply the postcode string seems to affect how it is matched.

from spacy.pipeline import EntityRuler
import en_core_web_lg

nlp = en_core_web_lg.load()

patterns = [
    {
        "label": "POST_CODE", "pattern": [
            {"TEXT": {"REGEX": r'^[a-zA-Z]{2}\d{1,2}[a-zA-Z]?$'}}, 
            {"TEXT": {"REGEX": r'^\d[a-zA-Z]{2}$'}}
            ]
        }
    ]
ruler = nlp.add_pipe("entity_ruler", config={"validate": True})
ruler.add_patterns(patterns)


def get_entities(text):
    entTypes = []
    doc = nlp(text)
    for entity in doc.ents:
        x = {}
        x[entity.label_] = entity.text
        entTypes.append(x)

    return {"entities": entTypes}


if __name__ == '__main__':
    import sys
    print(get_entities(sys.argv[1]))

Samples:

python test.py "this is a test RH15 9XR"
{'entities': [{'CARDINAL': '9XR'}]}

python test.py "this is a test RH15 9XR, RH4 9GG"
{'entities': [{'CARDINAL': '9XR'}, {'POST_CODE': 'RH4 9GG'}]}

python test.py "this is a test RH15 9XR, RH4 9GG, RH5 0XE"
{'entities': [{'CARDINAL': '9XR'}, {'POST_CODE': 'RH4 9GG'}, {'POST_CODE': 'RH5 0XE'}]}

python test.py "this is a test RH15 9XR, RH4 9GG, RH5 0XE PO01 0IO"
{'entities': [{'CARDINAL': '9XR'}, {'POST_CODE': 'RH4 9GG'}, {'CARDINAL': '0XE'}, {'CARDINAL': '0IO'}]}

It would appear that for some reason the first postcode is never matched. In the last example both "RH5 0XE" & "PO01 0IO" should match, as you can see "RH5 0XE" matched in the previous run.

Any ideas on how I can improve the match rate?

Thanks

Mark

user_461847 · Answer

Put the entity_ruler before ner in the pipeline so that its matches have priority over the CARDINAL spans from ner, or alternatively you can set it to overwrite overlapping entities with the overwrite_ents setting.

user_4755832 · Answer

If you run the regex with re instead of through spacy does it work?

I think you are not handling white spaces or end of string / start of string situations correctly in your regex -- all the examples where your test works have commas nearby while the examples failing have less commas around them

pattern match issue with SpaCy

2 Answers