Adding new language to spaCy: lemma lookup problem

Viewed 154

I'm writing a new spacy language class for ancient Greek (grc) and it is working at the tokenizer level but fails to do the lemma lookup in a dictionary I built from a corpus. When I run the test for the lookup table with python -m pytest spacy_lookups_data in the corresponding directory of spacy_lookups_data and with a test file I wrote, I get the following error:

grc_nlp = <spacy.lang.grc.AncientGreek object at 0x7fe8a8696730>, string = 'μοι', lemma = ' ἐγώ'

@pytest.mark.parametrize(
    "string,lemma",
    [
        ("ἄνδρα", " ἀνήρ"),
        ("μοι", " ἐγώ"),
        ("πολύτροπον", "πολύτροπονος"),
        ("ὃς", "ὃς"),
        ("πολλὰ", "πολύς"),
        ("Τροίης", "Τροία"),
    ],
)
def test_grc_lemmatizer_lookup_assigns(grc_nlp, string, lemma):
    tokens = grc_nlp(string)
  assert tokens[0].lemma_ == lemma

E AssertionError: assert 'μοι' == ' ἐγώ' E - ἐγώ E + μοι

test_grc.py:26: AssertionError _ test_grc_lemmatizer_lookup_assigns[\u03c0\u03bf\u03bb\u03cd\u03c4\u03c1\u03bf\u03c0\u03bf\u03bd-\u03c0\u03bf\u03bb\u03cd\u03c4\u03c1\u03bf\u03c0\u03bf\u03bd\u03bf\u03c2] _

I get similar errors for all 5 lemmas I wrote into the test file. Could this be an error related to unicode? Maybe the Greek polytonic characters are missing from spacy character definition? I suspect this because ancient use diacritics that may not be supported by spacy. Or is this error rather pointing to a problem with my lemma table?

Thanks.

Jacobo

1 Answers

It's probably not a unicode problem. Did you add the entry points in setup.cfg and spacy_lookups_data/__init__.py and reinstall the spacy-lookups-data package?

Related