Computing metrics per-class for imbalanced data

[AnselmC]

Before submitting

• ✅ Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• ✅ Did you read the contributor guideline, Pull Request section?
• ✅ Did you make sure to update the docs?
• ✅ Did you write any new necessary tests?

What does this PR do?

Draft PR as discussed in #174.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @AnselmC! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-06-14 12:17:10 UTC

[Borda]

@AnselmC how is it going here? ready for review?

[AnselmC]

@Borda Hi, so I believe it's at least ready for an initial discussion.
One point I was struggling with/unsure about is how to handle multi-dimensional multi-class data (and their different accumulations). As I understand (please correct me if I'm mistaken), this means that a sample may belong to multiple classes encoded by a multi-dimensional binary vector. Hence, when mdmc_average="global", these multi-dim vectors need to be treated as a label - this seems analogous to how I've dealt with integer labels.
In the samplewise case, however, each sample is treated as a separate "batch" and then the metrics are computed per "batch" and averaged - I'm not sure how to correctly address this but I'm inclined to keep functionality as is for this case.

Would appreciate some input on this or pointers to more documentation for these scenarios.

[codecov]

Codecov Report

Merging #204 (5d7a937) into master (ad5e360) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #204      +/-   ##
==========================================
+ Coverage   96.78%   96.82%   +0.03%     
==========================================
  Files          94       94              
  Lines        3084     3115      +31     
==========================================
+ Hits         2985     3016      +31     
  Misses         99       99              

Flag	Coverage Δ
Linux	78.70% <70.17%> (-0.31%)	⬇️
Windows	78.70% <70.17%> (-0.31%)	⬇️
cpu	78.70% <70.17%> (-18.02%)	⬇️
gpu	96.78% <100.00%> (+0.03%)	⬆️
macOS	78.70% <70.17%> (-18.02%)	⬇️
pytest	96.82% <100.00%> (+0.03%)	⬆️
python3.6	?
python3.8	?
python3.9	?
torch1.3.1	?
torch1.4.0	?
torch1.8.1	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
torchmetrics/classification/accuracy.py	96.15% <ø> (ø)
torchmetrics/classification/f_beta.py	100.00% <100.00%> (ø)
torchmetrics/classification/precision_recall.py	100.00% <100.00%> (ø)
torchmetrics/classification/specificity.py	100.00% <100.00%> (ø)
torchmetrics/classification/stat_scores.py	100.00% <100.00%> (ø)
torchmetrics/functional/classification/accuracy.py	94.36% <100.00%> (+0.42%)	⬆️
torchmetrics/functional/classification/f_beta.py	100.00% <100.00%> (ø)
...rics/functional/classification/precision_recall.py	100.00% <100.00%> (ø)
...chmetrics/functional/classification/specificity.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad5e360...5d7a937. Read the comment docs.

[SkafteNicki]

overall lgtm

[AnselmC]

@Borda sorry, I was out on vacation for a week. I fixed the valid issues from deep source but not sure how to deal with torch.tensor is not callable (see here)).
Anything else you need from me on this atm?

[Borda]

@SkafteNicki @maximsch2 mind review?

[SkafteNicki]

Great job, LGTM!
Please add entry to changelog on which metrics are affected by this enchancement :]

[AnselmC]

@SkafteNicki done!