Update class metrics interface of Precision/Recall/Fbeta to allow calculate them for each individual class

[junwen-austin]

🚀 Feature

I'd like to propose to update class metrics interface of Precision/Recall/Fbeta to have the average argument include none and weighted as in the corresponding functional metrics interface.

Motivation

Current interface with average argument restricts to macro and micro, and because of that one could not use class metrics interface to calculate precision/recall/fbeta for an individual class. For example, in binary classification, one is typically interested in getting metrics results for positive class (class 1) and this cannot be done with the current class interface. Therefore one has go back to the functional metric and this could defeat the purpose of having class metrics (to take care of ddp sync).

On the contrary, sklearn defaults to calculate precision/recall/fbeta for the individual class (class 1) while giving one option to calculate micro/macro/weighted average of these scores.
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html

Pitch

Update class metrics interface of Precision/Recall/Fbeta to have the average argument include none and weighted as in the corresponding functional metrics interface.

Alternatives

One can always fall back to the functional metric but I assume this is not what we would like.

Additional context

Really like the new class interface to work with DDP and appreciate all your work!

[SkafteNicki]

The Fbeta metric will get that feature when this PR is merged:
Lightning-AI/pytorch-lightning#4656
When that is merged I plan on dealing with precision and recall

[junwen-austin]

@SkafteNicki Thanks for the update. Any idea approximately when this feature (including for precision/recall/Fbeta) would be available?

[SkafteNicki]

I was hoping to get it all done for the 1.1 release :]

[shijianjian]

Got one question here.

For code like:
https://github.com/SkafteNicki/pytorch-lightning/blob/9d5b505f90e8d622a033666c3dbe2f0cc6ee5f26/pytorch_lightning/metrics/classification/precision_recall.py#L199-L205

It is computing the one-hot vector, which means to convert [0, 1] to [[1, 0], [0, 1]]. While in a binary case, it would put both negative and positive values into account (as true_positives as written in the code). So, I would say it will produce the wrong results for binary classification, right?

This issue somehow reported in Lightning-AI/pytorch-lightning#3035, but the reduction='none' is remaining unimplemented for the Precision and Recall classes.

I think an easy implementation of it is to have another param to set pos_labels=[0], which can receive n labels as well. I think it can be pretty helpful for both binary and multi-class classification.