The documentation with the algorithm used for training a NER model in spacy is not yet implemented. The only information provided is:
- that both the tagger, parser and entity recognizer(NER) using linear model with weights learned using the averaged perceptron algorithm.
- conjuction features out of atomic predictors are used to train the model.
These info are taken from: spacy-training-doc
Now, I have a trained a model with a new entity type(lets say animal) and reasonable high number of examples (>10000). Now trying some examples in the trained model I have:
input text: "cat is an animal" => cat: animal
input text: "tv is an animal" => tv: O
If average perceptron is used to evaluate a word as an entity shouldn't these two examples give the same results?(cat:animal, tv:animal) or is something that I am confused?
From my understanding the algorithm is using “gazetteer” features (lookup of entity names in a pre-compiled list created by the provided examples). If this is the case is there any way to exclude gazetteer features?
Any insights would be usefull.