SpaCy evaluation result is either 0 or 1

Viewed 115

I run the following code to train my NER model on the training set (75%) and evaluate on the validation set (25%) for a total of 20 iterations.

other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
val_score = []
with nlp.disable_pipes(*other_pipes):  # only train NER
    optimizer = nlp.begin_training()
    for itn in range(n_iter):
        print(itn, '/', n_iter)
        random.shuffle(train_doc)
        losses = {}
        nlp.update(
            [Example.from_dict(nlp(text), annotations) for text, annotations in tqdm(train_doc)],
            drop=0.5,  
            sgd=optimizer,
            losses=losses)
        curr_score = nlp.evaluate( [Example.from_dict(nlp.make_doc(text), annotations) for text, annotations in val_doc])
        val_score.append(curr_score)
        print(losses, curr_score)

However, the evaluation result doesn't look very normal. It's either 0 or 1, and it feels like my model is not really making any predictions. What went wrong? Thanks a lot.

Following is one of the results of an iteration:

{'ner': 3545.82652258873} {'token_acc': 1.0, 'token_p': 1.0, 'token_r': 1.0, 'token_f': 1.0, 'ents_p': 0.0, 'ents_r': 0.0, 'ents_f': 0.0, 'ents_per_type': {'location': {'p': 0.0, 'r': 0.0, 'f': 0.0}, 'group': {'p': 0.0, 'r': 0.0, 'f': 0.0}, 'creative-work': {'p': 0.0, 'r': 0.0, 'f': 0.0}, 'person': {'p': 0.0, 'r': 0.0, 'f': 0.0}, 'corporation': {'p': 0.0, 'r': 0.0, 'f': 0.0}, 'product': {'p': 0.0, 'r': 0.0, 'f': 0.0}}, 'speed': 29769.152836530662} 12 / 20

0 Answers
Related