Adding Span Categorizer to Spacy at the end of pipeline is not working

Fiesty_Anywhere 0
Viewed 46

I want to add spancat using default model at the end of my pipeline as shown.

nlp = spacy.load("en_core_web_sm")
from spacy.pipeline.spancat import DEFAULT_SPANCAT_MODEL
config = {
    "threshold": 0.5,
    "spans_key": "labeled_spans",
    "max_positive": None,
    "model": DEFAULT_SPANCAT_MODEL,
    "suggester": {"@misc": "spacy.ngram_suggester.v1", "sizes": [1, 2, 3]},
}
spancat = nlp.add_pipe("spancat", config=config)
optimizer = nlp.create_optimizer()
spancat.add_label("SPANCAT")
optimizer = nlp.initialise()

This gives me the following error

ValueError: [E955] Can't find table(s) lexeme_norm for language 'en' in spacy-lookups-data. Make sure you have the package installed or provide your own lookup tables if no default lookups are available for your language.

Why is this happening? I have read that for text categorizer training, you need to start with blank("en"), but I want to use "en_core_web_sm". What should I do? I want result as follows

spacy python
1 Answers

You do not want to do this. Calling nlp.initialize() on the whole pipeline will reset all the components in the pipeline including existing components like the tagger, and reinitializing everything is why it's trying to reload a lookups table that's already saved as part of en_core_web_sm.

Instead, follow the same steps described in https://stackoverflow.com/a/73659974, except without -o accuracy so that you train a component without vectors to be able to add it to en_core_web_sm:

  1. Convert your data with all spans saved under doc.spans["sc"]

  2. Create a config with spacy init config -p spancat

  3. Train

  4. Add the new spancat component to en_core_web_sm after replacing the spancat tok2vec listener with an internal tok2vec

    nlp_spancat = spacy.load("spancat_model")
nlp_spancat.replace_listeners("tok2vec", "spancat", ["model.tok2vec"])
nlp_combined = spacy.load("en_core_web_sm")
nlp_combined.add_pipe("spancat", source=nlp_spancat)
nlp_combined.to_disk("combined_model")

To make sure that your spancat component has the exact same base settings as en_core_web_sm (which includes the lexeme_norm lookups table from the error above), you can use this callback in your spancat config:

https://spacy.io/api/top-level#copy_from_base_model

[initialize.before_init]
@callbacks = "spacy.copy_from_base_model.v1"
tokenizer = "en_core_web_sm"
vocab = "en_core_web_sm"
aab0
Related