spaCy Tokenizer LEMMA and ORTH Exceptions Not Working

user_2917241 0
Viewed 1283

I'm following an example from Chapter #2 in the book: Natural Language Processing with Python and spaCy by Yuli Vasiliev 2020

enter image description here

The example is suppose to produce the lemmatization output:

['I', 'am' , 'flying' , 'to', 'Frisco']

['-PRON-', 'be' , 'fly' , 'to', 'San Francisco']

I get the following error:

nlp.tokenizer.add_special_case(u'Frisco', sf_special_case)
  File "spacy\tokenizer.pyx", line 601, in spacy.tokenizer.Tokenizer.add_special_case
  File "spacy\tokenizer.pyx", line 589, in spacy.tokenizer.Tokenizer._validate_special_case
ValueError: [E1005] Unable to set attribute 'LEMMA' in tokenizer exception for 'Frisco'. Tokenizer exceptions are only allowed to specify ORTH and NORM.

Could someone please advise for a workaround? I'm not sure if SpaCy version 3.0.3 was changed to no longer allow LEMMA to be part of tokenizer exception? Thanks!

spacy nlp
2 Answers

See https://github.com/explosion/spaCy/issues/7014

import spacy

nlp = spacy.load('en_core_web_sm')

nlp.get_pipe("attribute_ruler").add([[{"TEXT": "Frisco"}]], {"LEMMA": "San Francisco"})

doc = nlp(u'I am flying to Frisco and after to frisco')    
print(['token:%s lemma:%s' % (t.text, t.lemma_) for t in doc])
user_157169090

Try in Google Colab. The code works fine as it uses version 2.2.4 of spacy.

user_139109190
Related