accuracy, recall, precision and f1-score are equal

[Lucienxhh]

🐛 Bug

@Borda thanks for your advice.
i follow your suggestion in #543 that setting the average as 'macro'. however, it doesn't work.
i @ u in #1111, but you have not replied yet. i try to re-open #1111 , but fail.
therefore, i have to create a new issue for discussion.

To Reproduce

here is the code that compares torchmetrics and sklearn.metrics

import torch
import torchmetrics
from sklearn import metrics
torch.manual_seed(4)
batches = 10
num_classes = 2

# torchmetric init
average = 'macro' if num_classes == 2 else 'micro'
torchmetrics_accuracy = torchmetrics.Accuracy()
torchmetrics_recall = torchmetrics.Recall(average=average, num_classes=num_classes)
torchmetrics_precision = torchmetrics.Precision(average=average, num_classes=num_classes)
torchmetrics_f1 = torchmetrics.F1Score(average=average, num_classes=num_classes)

pred = []
real = []
for i in range(batches):
    preds = torch.randn(100, num_classes).softmax(dim=-1)
    target = torch.randint(2, (100,))
    
    # torchmetrics
    torchmetrics_accuracy(preds, target)
    torchmetrics_precision(preds, target)
    torchmetrics_recall(preds, target)
    torchmetrics_f1(preds, target)
    
    # sklearn.metrics
    preds_ = preds.cpu().numpy()
    idx = preds_.argmax(axis=1)
    pred.extend(idx.tolist())
    real.extend(target.cpu().numpy().tolist())


# calculation of torchmetrics
acc_1 = torchmetrics_accuracy.compute()
precision_1 = torchmetrics_precision.compute()
recall_1 = torchmetrics_recall.compute()
f1_1 = torchmetrics_f1.compute()
print(acc_1, recall_1, precision_1, f1_1)

# calculation of sklearn.metrics
average = 'binary' if num_classes == 2 else 'micro'
acc_2 = metrics.accuracy_score(real, pred)
precision_2 = metrics.precision_score(real, pred, average=average)
recall_2 = metrics.recall_score(real, pred, average=average)
f1_2 = metrics.f1_score(real, pred, average=average)
print(acc_2, recall_2, precision_2, f1_2)

the results differ, which makes me confused.

Expected behavior

i have read the doc of torchmetrics and sklearn.metrics, and found that torchmetrics didn't have binary option for average.
details are here: sklearn.metrics-doc and torchmetrics-doc

[SkafteNicki]

Hi @Lucienxhh,
We are aware of this problem when evaluating the metrics in for binary problems and are in the process of a larger refactor of the classification package (#1001)
In the mean time if you want to match the results of sklearn you can do something like this:

torchmetrics_recall = torchmetrics.Recall(average=None, num_classes=num_classes)  # return score for both classes
torchmetrics_recall(preds, target)[-1]  # only extract the positive which corresponds to sklearns

[Lucienxhh]

ok, the issue is closing.