MinMaxMetric for wrapping other metrics

[janhenriklambrechts]

🚀 Feature

Motivation

• MinMaxMetric is a metric that simply wraps another metric (e.g.val_acc) and creates a new metric that tracks the min, max or both values of val_acc.

Pitch

• I personally use it to quickly see the max_val_acc of a complete experiment in TensorBoard (instead of going through the graph manually to find the max value) but I can see other usecases as well.
• It was discussed in the PL Slack here and clearly resonated with more users

Additional context

• Happy to submit a PR for this feature, as I do have already a (incomplete) MaxMetric code implemented here

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Hi @janhenriklambrechts are you still interested in sending a PR for this feature?

[janhenriklambrechts]

Absolutely, you can assign it to me.
Think I will be able to start working on it in a few days :)

[SkafteNicki]

@janhenriklambrechts Nice :]
Feel free to ping if you have any question (here or on slack).
If you need inspiration, then I have a open PR for another wrapper metric (#101) for creating bootstrapped versions of metrics.

[janhenriklambrechts]

Thank you so much for the inspiration! Very useful!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[janhenriklambrechts]

Will only be able to work on it from beginning of July onward @SkafteNicki - is that OK?

[Borda]

Will only be able to work on it from beginning of July onward @SkafteNicki - is that OK?

that is great! looking forward 🐰

[SkafteNicki]

@janhenriklambrechts any updates on this? :]

[janhenriklambrechts]

Hi @SkafteNicki, completely lost track of this - I am starting this weekend! Thanks for pinging me :)

[SkafteNicki]

@janhenriklambrechts no problem. Let me know if you need any help :]

[nils-werner]

Right now I can't log() this if it is inside a MetricCollection(), right?

self.val_metric = torchmetrics.MetricCollection({
    'acc': torchmetrics.Accuracy(),
    'acc_best': torchmetrics.MinMaxMetric(torchmetrics.Accuracy())
})

because .compute() of the collection will return a nested dict

{'acc': tensor(0.9497), 'acc_best': {'raw': tensor(0.9595), 'max': tensor(1.), 'min': tensor(0.8281)}}

and I get an error message

ValueError: `self.log(val, {'acc': tensor(0.9497), 'acc_best': {'raw': tensor(0.9595), 'max': tensor(1.), 'min': tensor(0.8281)}})` was called, but nested dictionaries cannot be logged

It would be great if we could have a MaxMetric() wrapper that simply returns the max value, so that .compute() returns

{'acc': tensor(0.9497), 'acc_best': tensor(1.)}