Wrong numbers for multiclass acc/prec/f1 if some class is not listed in true/pred tensors

[notonlyvandalzzz]

🐛 Bug

TM's output for precision, accuracy, f1 differs from sklearn' version of same metrics with average='macro' setting used and labels in pred/true tensors for all classes except one

To Reproduce

Steps to reproduce the behavior:

1. Get one tensor with classlabels as y_pred, copy it to y_true, and pull thru torchmetrics.functional.f1/accuracy/precision with average="macro", num_classes=number of your classes used for that.
2. Convert that tensors to lists and pass into sklearn metrics, of course with average=macro where its possible
3. Compare numbers

Code sample

y_pred = y_true = torch.tensor([1, 6, 6, 6, 3, 6, 3, 6, 6, 3, 6, 3, 6, 6, 3, 6, 3, 3, 6, 6, 6, 6, 6, 6,
        6, 3, 5, 6, 6, 3, 6, 6, 6, 6, 3, 6, 6, 3, 3, 1, 3, 1, 6, 3, 3, 1, 3, 6,
        4, 6, 6, 6, 6, 6, 6, 3, 3, 6, 3, 6, 1, 0, 5, 3, 6, 6, 6, 6, 3, 0, 6, 3,
        3, 3, 6, 3, 4]) # no class with id=2 !
print(f'Metrics:')
print(f'F1: {f1_score(y_true.numpy().astype(int).tolist(), y_pred.numpy().astype(int).tolist(), average="macro")}')
print(f'ACC: {accuracy_score(y_true.numpy().astype(int).tolist(), y_pred.numpy().astype(int).tolist())}')
print(f'PREC: {precision_score(y_true.numpy().astype(int).tolist(), y_pred.numpy().astype(int).tolist(), average="macro")}')
print(f'TM F1: {torchmetrics.functional.f1(y_pred, y_true, average="macro", num_classes=7)}')
print(f'TM acc: {torchmetrics.functional.accuracy(y_pred, y_true, average="macro", num_classes=7)}')
print(f'TM prec: {torchmetrics.functional.precision(y_pred, y_true, average="macro", num_classes=7)}')

Metrics:
F1: 1.0
ACC: 1.0
PREC: 1.0
TM F1: 0.8571428656578064
TM acc: 0.8571428656578064
TM prec: 0.8571428656578064

Expected behavior

They're should be 1.0, similar to sklearn version
But adding even single item with id=2 makes numbers right, so its definitely a bug when label is exist but no examples for it in pred and true tensors

Environment

• CUDA:
  - GPU:
  - NVIDIA Quadro RTX 5000
  - available: True
  - version: 11.0
• Packages:
  - numpy: 1.20.3
  - pyTorch_debug: True
  - pyTorch_version: 1.7.0+cu110
  - pytorch-lightning: 1.3.4
  - torchmetrics: 0.3.2
  - tqdm: 4.51.0
• System:
  - OS: Linux
  - architecture:
  - 64bit
  -
  - processor: x86_64
  - python: 3.7.10
  - version: Add metric code from lightning #1 SMP Sun Feb 14 18:10:38 EST 2021

[github-actions]

Hi! thanks for your contribution!, great first issue!

[vatch123]

Hi @Borda I can have a look at this if someone is already not involved here.

[Borda]

Hi @Borda I can have a look at this if someone is already not involved here.

That would be great, thank you!

[SkafteNicki]

Hi @vatch123
Thank you for wanting to contribute. Just wanted to let you know that I debugged this a bit I think it can be solved by changing this line
https://github.com/PyTorchLightning/metrics/blob/1841cad3839f5d1907a1bb8bb6a266de5c5333f9/torchmetrics/functional/classification/stat_scores.py#L186
to also account for the zero_div_mask so

weights = torch.where(zero_div_mask | ignore_mask, tensor(0.0, device=weights.device), weights)

[vatch123]

Hi @vatch123
Thank you for wanting to contribute. Just wanted to let you know that I debugged this a bit I think it can be solved by changing this line
https://github.com/PyTorchLightning/metrics/blob/1841cad3839f5d1907a1bb8bb6a266de5c5333f9/torchmetrics/functional/classification/stat_scores.py#L186

to also account for the zero_div_mask so

weights = torch.where(zero_div_mask | ignore_mask, tensor(0.0, device=weights.device), weights)

Hi. Thanks for this. Let me have a look at this.

[celsofranssa]

Any update on this?

[SkafteNicki]

@celsofranssa we have a open PR #303 that should fix it.