Calculate precision, recall, f1 score for custom dataset for multiclass classification Huggingface library

user_14562845 0
Viewed 458

I am trying to do multiclass classification for the sentence pair task. I uploaded my custom dataset of train and test separately in the hugging face data set and trained my model and tested it and was trying to see the f1 score and accuracy.

I tried

from datasets import load_metric
metric1 = load_metric("precision")
metric2 = load_metric("recall")
metric3 = load_metric("f1")
metric = load_metric('accuracy')

def compute_metrics(eval_pred):
   metric1 = load_metric("precision")
   metric2 = load_metric("recall")
   metric3 = load_metric("f1")
   metric4 = load_metric("accuracy")

   logits, labels = eval_pred
   predictions = np.argmax(logits, axis=-1)
   precision = metric1.compute(predictions=predictions, references=labels,average ="micro")["precision"]
   recall = metric2.compute(predictions=predictions, references=labels,average ="micro")["recall"]
   f1 = metric3.compute(predictions=predictions, references=labels,average ="micro")["f1"]
   accuracy = metric4.compute(predictions=predictions, references=labels,average ="micro")["accuracy"]
   return {"precision": precision, "recall": recall, "f1": f1, "accuracy": accuracy}

but it says

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

How can I fix this and print precision, recall, and f1 score?

huggingface-datasets huggingface-tokenizers bert-language-model huggingface-transformers python-3.x
1 Answers

Your code should have thrown a different error as metric compute should not have "average" parameter.

So, it looks like you might have forgotten to reinitialize your trainer with the corrected definition of compute_metrics.

Here's the code that works:

def custom_metrics(eval_pred):
    metric1 = load_metric("precision")
    metric2 = load_metric("recall")
    metric3 = load_metric("f1")
    metric4 = load_metric("accuracy")
    
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)

    precision = metric1.compute(predictions=predictions, references=labels, average="micro")["precision"]
    recall = metric2.compute(predictions=predictions, references=labels, average="micro")["recall"]
    f1 = metric3.compute(predictions=predictions, references=labels, average="micro")["f1"]
    accuracy = metric4.compute(predictions=predictions, references=labels)["accuracy"]

    return {"precision": precision, "recall": recall, "f1": f1, "accuracy": accuracy}


trainer = Trainer(model, args, train_dataset=dds['train'], eval_dataset=dds['test'],
                          tokenizer=tokenizer, compute_metrics=custom_metrics)

trainer.train()
user_41600300
Related