Is there a way to evaluate losses on the test sample using spacy model

user_5046464 0
Viewed 1216

I am trying to create a binary classifier with spacy 2.1.3 and in order to perform an overfitting test I would like to evaluate losses on the test sample. In their tutorial losses are used as a parameter and somehow updated:

https://github.com/explosion/spaCy/blob/master/examples/training/train_textcat.py#L90

I cannot find any example of how to evaluate it on my test sample. Ideally I would like to produce plots as show here:

https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/

I tried digging into their code but I didn't find anything useful. Has anyone tried to produce similar plots?

Thank you for your help and comments :)

text-classification loss-function spacy nlp
1 Answers

The variable losses is being set during the training loop, cf. https://github.com/explosion/spaCy/blob/master/spacy/pipeline/pipes.pyx#L931.

What you want to do, is after each iteration (epoch), print out this training loss, but also perform your own evaluation on a held-out dev test set. When you apply your model-in-training to the dev set, you can use average model parameters as explained here: https://spacy.io/usage/training#tips-param-avg.

For this dev evaluation, you can implement whatever metric you like, such as accuracy, precision, recall, F-score, or a loss function similar to the one you've been training on, cf. https://github.com/explosion/spaCy/blob/master/spacy/pipeline/pipes.pyx#L950.

user_79618600
Related