You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using a compositional metric, i.e., metric_a + metric_b or 2 * metric_a with forward() instead of update(), the value returned by .compute() will be based only on the last batch.
This is because the default implementation of Metric.forward(), which is inherited by CompositionalMetric, calls reset(), which in turn resets all child metrics. Their state, however, is not restored by forward() afterwards.
My proposed fix would be to override forward() in CompositionalMetric and I will file a PR implementing this.
To Reproduce
Code sample
importtorchimporttorchmetricsfromtorchmetrics.metricimportCompositionalMetric# Example predictions and targetspred=torch.tensor([0.4, 0.2, 0.6, 0.7, 0.5])
target=torch.tensor([0.5, 0.4, 0.4, 0.3, 0.6])
# Compute base metric in batchesmetric=torchmetrics.MeanSquaredError()
metric(preds=pred[:3], target=target[:3])
metric(preds=pred[3:], target=target[3:])
val=metric.compute()
print(val)
# Compute compositional metric in batchescm=1*torchmetrics.MeanSquaredError()
cm(preds=pred[:3], target=target[:3])
cm(preds=pred[3:], target=target[3:])
val=cm.compute()
print(val)
Expected behavior
Two identical values are printed:
tensor(0.0520)
tensor(0.0520)
Actual (faulty) behavior
The compositional metric only returns the value for the last batch:
tensor(0.0520)
tensor(0.0850)
Environment
PyTorch Version (e.g., 1.0): 1.10.0
OS (e.g., Linux): Linux
How you installed PyTorch (conda, pip, source): conda
Build command you used (if compiling from source):
Python version: 3.8
CUDA/cuDNN version: 11.1
GPU models and configuration:
Any other relevant information:
Additional context
The text was updated successfully, but these errors were encountered:
🐛 Bug
When using a compositional metric, i.e.,
metric_a + metric_b
or2 * metric_a
withforward()
instead ofupdate()
, the value returned by.compute()
will be based only on the last batch.This is because the default implementation of
Metric.forward()
, which is inherited byCompositionalMetric
, callsreset()
, which in turn resets all child metrics. Their state, however, is not restored byforward()
afterwards.My proposed fix would be to override
forward()
inCompositionalMetric
and I will file a PR implementing this.To Reproduce
Code sample
Expected behavior
Two identical values are printed:
Actual (faulty) behavior
The compositional metric only returns the value for the last batch:
Environment
conda
,pip
, source): condaAdditional context
The text was updated successfully, but these errors were encountered: