I am working with a dataset of ~90000 tokens (not huge) and in this dataset only ~600 are labelled with something other than the default (not interesting) label (So, in the IOB format more than 89000 tokens are labelled 'O', the rest is labelled with one of five different categories, I'd like to learn).
I am using a bi-lstm network that performs nicely on a sample data set where the 'interesting' categories are relatively more frequent (some 5000 of 12000 tokens are non-'O').
But when training the same network on my data, the outcome is that practically no non-'O' tokens are predicted - it seems that the rare non-'O'events are just not frequent enough for the NN to give them a chance.
In that situation I was thinking about a custom loss function penalizing missing a non-'O' token more than vice versa (falsely predicting a 'O' as a non-'O').
Are there any other best practices that are recommended? (600 out of 90000 doesn't seem too rare, but still, currently I don't get any non-'O' predictions.)
The model I am using is Adam Atkinson's nice implementation (https://github.com/aatkinson/deep-named-entity-recognition) of a keras model which as I said works nicely on a comparable dataset with less rare entities.