'torchmetrics' does not work with PyTorchLightning

Question

I am trying to understand how to use torchmetrics with PyTorch Lightning.
But, I got a same output with Accuracy, F1-score, Precision, etc.

This is the code.

metric_acc = torchmetrics.Accuracy()
metric_f1 = torchmetrics.F1()
metric_pre = torchmetrics.Precision()
metric_rec = torchmetrics.Recall()

n_batches = 3
for i in range(n_batches):
    # simulate a classification problem
    preds = torch.randn(10, 5).softmax(dim=-1)
    target = torch.randint(5, (10,))

    acc = metric_acc(preds, target)
    f1 = metric_f1(preds, target)
    pre = metric_pre(preds, target)
    rec = metric_rec(preds, target)
    print(f"Accuracy on batch {i}: {acc}")
    print(f"F1 score on batch {i}: {f1}")
    print(f"pre score on batch {i}: {pre}")
    print(f"rec score on batch {i}: {rec}")
    print('-' * 20)


acc = metric_acc.compute()
f1 = metric_f1.compute()
pre = metric_pre.compute()
rec = metric_rec.compute()
print(f"Accuracy on all data: {acc}")
print(f"f1 score on all data: {f1}")
print(f"pre score on all data: {pre}")
print(f"rec score on all data: {rec}")

Result is here.

Accuracy on batch 0: 0.10000000149011612
F1 score on batch 0: 0.10000000894069672
pre score on batch 0: 0.10000000149011612
rec score on batch 0: 0.10000000149011612
--------------------
Accuracy on batch 1: 0.30000001192092896
F1 score on batch 1: 0.30000001192092896
pre score on batch 1: 0.30000001192092896
rec score on batch 1: 0.30000001192092896
--------------------
Accuracy on batch 2: 0.4000000059604645
F1 score on batch 2: 0.40000003576278687
pre score on batch 2: 0.4000000059604645
rec score on batch 2: 0.4000000059604645
--------------------
Accuracy on all data: 0.2666666805744171
f1 score on all data: 0.2666666805744171
pre score on all data: 0.2666666805744171
rec score on all data: 0.2666666805744171

Process finished with exit code 0

I got the same result when I used it with PyTorchLightning, so I try it with simple code and got the same thing.
If you know the problem or the solution, please let me know.
Thank you very much.

user_4521646 · Answer

I guess that you are looking for this simple example of how to use TorchMetrics for logging your training progress with PytorchLightning. Otherwise, could you more elaborate on your use-case, ideally add your application example?

user_2596276 · Answer

The reason for this is that for multi class classification if you are using F1, Precision, ACC and Recall with micro (the default )these are equivalent metrics and recommending you should use macro

metric_acc = torchmetrics.Accuracy(average='macro')
metric_f1 = torchmetrics.F1(average='macro')
metric_pre = torchmetrics.Precision(average='macro')
metric_rec = torchmetrics.Recall(average='macro')

'torchmetrics' does not work with PyTorchLightning

2 Answers