I'm writing a new spacy language class for ancient Greek (grc) and it is working at the tokenizer level but fails to do the lemma lookup in a dictionary I built from a corpus. When I run the test for the lookup table with python -m pytest spacy_lookups_data in the corresponding directory of spacy_lookups_data and with a test file I wrote, I get the following error:
grc_nlp = <spacy.lang.grc.AncientGreek object at 0x7fe8a8696730>, string = 'μοι', lemma = ' ἐγώ'
@pytest.mark.parametrize(
"string,lemma",
[
("ἄνδρα", " ἀνήρ"),
("μοι", " ἐγώ"),
("πολύτροπον", "πολύτροπονος"),
("ὃς", "ὃς"),
("πολλὰ", "πολύς"),
("Τροίης", "Τροία"),
],
)
def test_grc_lemmatizer_lookup_assigns(grc_nlp, string, lemma):
tokens = grc_nlp(string)
assert tokens[0].lemma_ == lemma
E AssertionError: assert 'μοι' == ' ἐγώ' E - ἐγώ E + μοι
test_grc.py:26: AssertionError _ test_grc_lemmatizer_lookup_assigns[\u03c0\u03bf\u03bb\u03cd\u03c4\u03c1\u03bf\u03c0\u03bf\u03bd-\u03c0\u03bf\u03bb\u03cd\u03c4\u03c1\u03bf\u03c0\u03bf\u03bd\u03bf\u03c2] _
I get similar errors for all 5 lemmas I wrote into the test file. Could this be an error related to unicode? Maybe the Greek polytonic characters are missing from spacy character definition? I suspect this because ancient use diacritics that may not be supported by spacy. Or is this error rather pointing to a problem with my lemma table?
Thanks.
Jacobo