adding spacy infixes: works for uppercase, but not lowercase, parentheses but not slashes

user_1716733 0
Viewed 258

I am trying to make sure that spacy treats dot as a separate token except when it is between two digits. I noticed nlp.Defaults.infixes uses lookaround operators extensively, so I followed the example:

infixes = nlp.Defaults.infixes + (r'''[;,:]''',
                                  r'(?<=[a-zA-Z_])[\.^]',  r'[\.^](?=[a-zA-Z_])',
                                  ) 
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

Now upper case behaves as expected:

list(nlp.tokenizer('HELLO.WORLD'))
[HELLO, ., WORLD]

But if right-hand side is lowercase, it fails:

list(nlp.tokenizer('HELLO.world'))
[HELLO.world]

list(nlp.tokenizer('hello.world'))
[hello.world]

list(nlp.tokenizer('hello.WORLD'))
[hello, ., WORLD]

Another example, where a regex finds parentheses but not slashes:

infixes = nlp.Defaults.infixes + (r'(?<=[a-zA-Z_])[\.\(\)/](?=[a-zA-Z_])', ) 
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

tests = [
    'mid(inferior',
    'mid/inferior',
    'left.mid',
    'left.mid/inferior']

pattern = re.compile(r'(?<=[a-zA-Z_])[\.\(\)/](?=[a-zA-Z_])')
for tt in tests:
    print("-"*20)
    print(pattern.split(tt))
    print(list(nlp.tokenizer(tt)))

result:

--------------------
['mid', 'inferior']
[mid, (, inferior]
--------------------
['mid', 'inferior']
[mid, /, inferior]
--------------------
['left', 'mid']
[left.mid]
--------------------
['left', 'mid', 'inferior']
[left.mid/inferior]

As one can see, for some reason within spacy, the regex pattern does not split on dot, and if word is not purely alphabetic, it fails to split on slash, even though that should not be an issue for the above regex

spacy
1 Answers

The problem here is the order of the infix patterns. It uses the first match it finds and another infix pattern is matching before your pattern. You can add your custom patterns first instead:

infixes = (r'(?<=[a-zA-Z_])[\.\(\)/](?=[a-zA-Z_])', ) + nlp.Defaults.infixes

This can potentially lead to side effects for other patterns, especially if you add very short patterns first, so double-check the overall tokenization accuracy for your data before and after making the changes to make sure it's doing what you expect. This is probably fine as-is, but you might need to add your pattern in the middle rather than at the beginning to get the intended results.

Edit:

This doesn't work because the URL pattern is jumping in first.

print(nlp.tokenizer.explain("HELLO.world"))
# [('URL_MATCH', 'HELLO.world')]

If you don't care about URL tokenization, you can just set:

nlp.tokenizer.url_match = None

to remove the URL matching that's going to interact a lot with . and / in tokens.

user_4618470
Related