Spacy - Use two trainable components with two different datasets

user_14104547 0
Viewed 363

I was wondering if it is possible to train two trainable components in Spacy with two different datasets ? In fact, I would like to use the NER and the text classifier but since the train datasets for these two components should be annotated differently so I don't know how can I train both components at once...

Should I train each task in a separate pipeline and assemble both pipelines at the end ? Or should I train the NER, package this pipeline and then use this package as input to train the text classifier ?

Many thanks in advance for your help

spacy-3 spacy nlp
1 Answers

You won't be able to train these at the same time, if the dataset is not the same.

If you're working with spaCy v3, it should be relatively straightforward to combine the two training steps into one final pipeline. For instance, create a config that trains the NER first, and store it to disk. Then, create a new config where you source the NER from the previously trained pipeline, and then define this NER component as frozen:

[nlp]
pipeline = ["ner", "textcat"]
...

[training]
frozen_components = ["ner"]
...

[components.ner]
source = "your_trained_ner_location"
component = "ner"

[components.textcat]
factory = "textcat"
...

Now run training on your textcat.

FYI - this kind of multi-step workflows can be easily set-up with spacy projects

user_79618600
Related