Is there a way to call Macro-Precision in Hugging Face Trainer?

user_14605695 0
Viewed 49

I'm currently making tests on the DEFT-2015 dataset using Hugging Face models. I would like to compare my results to what has been done.

I checked in the list_metrics method from the datasets library, but I did not see Macro Precision, which was the metric used at the time by the researchers.

Do you have any indication for how I could tackle the problem ?

huggingface-datasets huggingface-transformers nlp
1 Answers

The huggingface library (version 4.20.0) seems to depend behind the curtains calls to scikit-learn library.

If you just use (without using scikit-learn):

from datasets import load_metric
metric = load_metric("precision")
precision = metric.compute(predictions=[...],references=[...])

it will throw an error that

scikit-learn is not installed.

Why this intro?

Well, in fact, you can easily use datasets metrics to calculate however you want your metric (just exactly like scikit-learn does).

You just need to add the 'average' parameter:

  from datasets import load_metric
  metric = load_metric("precision")
  precision = metric.compute(predictions = [0, 0, 0, 0, 1, 1, 2, 2],
                             references  = [0, 0, 0, 0, 1, 1, 1, 2],
                             average='macro')
  # This prints 0.833333334
  print(precision)

The snippet above will print {'precision': 0.8333333333333334}, because (1 + 1 + 0.5) / 3 = 0.83, which is exactly the definition of macro precision you are searching for.

Conclusion : Use the average parameter to set the way you want to calculate your metric (micro/macro/weighted).

user_61170170
Related