Calibration error does not support negative logits

[pietrolesci]

Hi there,

It appears that CalibrationError currently does not support some of the input data types mentioned in this table from the docs. In particular, it seems to break when fed with

• preds=logits where logits is a (N, C) float32 tensor with potentially negative values.
• preds=predictions where predictions is a (N,) int tensor with the predicted labels.

It still works with softmax-ed logits (preds=logits.softmax(-1)) or, generally, with and (N,)-dimensional float32 tensors (i.e. the binary input in the input data table mentioned above).

To reproduce:

N, C = 10, 3
targets = torch.randint(C, (N,))

# (N, C) non-negative: works
preds = torch.rand((N, C))  # non-negative
CalibrationError()(preds=preds, target=targets)

# (N, C) potentially negative: fails
preds = torch.randn((N, C))  # potetially negative
CalibrationError()(preds=preds, target=targets)

# (N,) int type: fails
CalibrationError()(preds=targets, target=targets)

# (N,) float type non-negative: works
preds = torch.rand((N,))  # binary non-negative
CalibrationError()(preds=preds, target=targets)

# (N,) float type potentially negative: fails
preds = torch.randn((N,))  # binary potentially negative
CalibrationError()(preds=preds, target=targets)

Torchmetrics: v0.8
Pytorch: v1.11

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Hi @pietrolesci,
I have now created PR #985 to fix the issue. One note is that the metric cannot the case where preds is an int tensor since the metric needs access to the raw probabilities/logits of the model.

[pietrolesci]

Hi @SkafteNicki,
Thanks for acting so quickly on this!