Accuracy metric does not work on Model predictions.

[ajsanjoaquin]

🐛 Bug

Error when using pytorch_lightning.metrics.Accuracy on logits, even though that is exactly the use-case stated in the docs.

To Reproduce

Steps to reproduce the behavior:

Using the accuracy metric on either the training or validation step with the expected logit inputs ((64 x 10), 10 = # of classes) and target vector (64) throws the error ValueError: The `preds` should be probabilities, but values were detected outside of [0,1] range. similar to #551.

Code sample

class ToyModel(pl.LightningModule):
  def __init__(self, batch_size, max_epochs = 10, learning_rate=0.01, num_classes=10, noise=0.6, alpha=0.1, beta=1):
    super(ToyModel, self).__init__()

    # Model
    self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
    self.conv1_bn = nn.BatchNorm2d(64)
    
    self.conv2 = nn.Conv2d(64, 128, 3, padding=1)
    self.conv2_bn = nn.BatchNorm2d(128) # pooling changes in_channel for next layer?

    self.conv3 = nn.Conv2d(128, 196, 3, padding=1)
    self.conv3_bn = nn.BatchNorm2d(196)

    self.fc1 = nn.Linear(in_features=3136, out_features=256)
    self.fc1_bn = nn.BatchNorm1d(256)

    self.fc2 = nn.Linear(256, num_classes)

    # Parameters
    self.alpha = alpha
    self.beta = beta
    self.noise = noise

    self.max_epochs = max_epochs
    self.learning_rate= learning_rate
    self.batch_size = batch_size

    # Accuracy Metric
    self.train_acc = pl.metrics.Accuracy()

  def loss(self, softmax_pred, target):
    return self.alpha * nn.CrossEntropyLoss()(softmax_pred, target) + self.beta * nn.CrossEntropyLoss()(softmax_pred, target)


  def forward(self, x):
    x = self.conv1(x)
    x = self.conv1_bn(x)
    x = F.relu(x)
    x = F.max_pool2d(x, kernel_size=2, stride=2)
    
    x = self.conv2(x)
    x = self.conv2_bn(x)
    x = F.relu(x)
    x = F.max_pool2d(x, kernel_size=2, stride=2)

    x = self.conv3(x)
    x = self.conv3_bn(x)
    x = F.relu(x)
    x = F.max_pool2d(x, kernel_size=2, stride=2)
    
    x = torch.flatten(x, start_dim=1)
    x = self.fc1(x)
    x = self.fc1_bn(x)
    x = F.relu(x)

    return self.fc2(x) 
    
  def training_step(self, batch, batch_idx):
    inputs, targets = batch
    predictions = self(inputs)
    loss = self.loss(predictions, targets)
    self.log("train_acc_step", self.train_acc(predictions, targets))
    return loss

  def training_epoch_end(self, outs):
    # log epoch metric
    self.log('train_acc_epoch', self.train_acc.compute())

Passing torch.tensor(64, 3, 32, 32) as training data should trigger the error

Expected behavior

self.train_acc(predictions, targets) should compute the accuracy

Environment

I used Google Colab

• PyTorch Version (e.g., 1.0): 1.8.0
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): pip
• Python version: 3.8
• CUDA/cuDNN version: 11.0

Additional context

Workaround: I followed the advice in #551 and reinstalled pytorch lightning to 1.1.8.

Since this Lightning version triggers #6210 in Colab, I also had to reinstall a different version of torch:

!pip install torchtext==0.8.0 torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html -q
!pip install pytorch-lightning==1.1.8 -q

[github-actions]

Hi! thanks for your contribution!, great first issue!

[mmxuan18]

meet the same problem when pytorch-lighting 1.2.0. when 1.1.6 run success

[rchen19]

I think this is because previous PL classification metrics does check if the preds are between 0 and 1 when they are floats, but just assumed they are probabilities, this could give you incorrect results without any warning.

Now PL actually checks the values of preds if they are floats, and enforces the range to be [0, 1], so it looks like you need to add a sigmoid layer after your network output, to make sure they are probabilities, not logits.

[mmxuan18]

I think this is because previous PL classification metrics does check if the preds are between 0 and 1 when they are floats, but just assumed they are probabilities, this could give you incorrect results without any warning.

Now PL actually checks the values of preds if they are floats, and enforces the range to be [0, 1], so it looks like you need to add a sigmoid layer after your network output, to make sure they are probabilities, not logits.

i solve this by add softmax to the logits, because pytorch crossentroyloss include softmax so the output not add this, but metris need

[akihironitta]

@SkafteNicki Could you have a look at this issue?

[SkafteNicki]

Our metrics interface does not currently support non-normalized inputs (e.g. logits), here is a open issue for it in the torchmetrics repo: Lightning-AI/torchmetrics#60.
I have it in my pipeline within the next couple of weeks, as I am in the process of standardizing the internals of metrics.

As @mlinxiang mentions, the solution for now is to apply softmax transformation to your input.

[dleve123]

@ajsanjoaquin I'm curious if just adding a softmax layer to your model did the trick to work lightning's metrics and led to a well-trained model? It seems like you shouldn't use nn.CrossEntropyLoss when your model outputs normalized probabilities.

What ended up working here?

[ajsanjoaquin]

@dleve123 Yup. I added the softmax layer and used negative log-likelihood loss instead.

[NicoMandel]

Hi all.
I came across this bug when trying to re-implement the Youtube masterclass tutorial - and I am not quite sure what the fix is... So in the video, the import is from pytorch_lightning.metrics.functional import accuracy, and the use is in the traininig step function... Would I have to redefine my model and redefine the loss function, or can I just put an extra step in here:

        # 2 computing the loss function
        J = self.loss(logits, y)

        # lightning 4 - also getting the accuracy and showing it in the pbar
        acc = accuracy(logits, y)
        pbar = {'train_acc': acc}

Sorry for the confusion, thanks for your help. Great library, really helpful for beginners like me
Cheers
Nico

[rchen19]

Hi all.
I came across this bug when trying to re-implement the Youtube masterclass tutorial - and I am not quite sure what the fix is... So in the video, the import is from pytorch_lightning.metrics.functional import accuracy, and the use is in the traininig step function... Would I have to redefine my model and redefine the loss function, or can I just put an extra step in here:

        # 2 computing the loss function
        J = self.loss(logits, y)

        # lightning 4 - also getting the accuracy and showing it in the pbar
        acc = accuracy(logits, y)
        pbar = {'train_acc': acc}

Sorry for the confusion, thanks for your help. Great library, really helpful for beginners like me
Cheers
Nico

Can you specify the error you are getting?

[NicoMandel]

Hi @rchen19 thanks for the comment. The error I am getting is the following:

Exception has occurred: ValueError
The `preds` should be probabilities, but values were detected outside of [0,1] range.

The code it is supposed to be running is:

logits = self(x)     # getting the logits out

        # 2 computing the loss function
        J = self.loss(logits, y)

        # lightning 4 - also getting the accuracy and showing it in the pbar
        acc = accuracy(logits, y)
        pbar = {'train_acc': acc}

in the training_step function.
The network is defined as:

def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(28 * 28, 64)
        self.l2 = nn.Linear(64, 64)
        self.l3 = nn.Linear(64, 10)
        self.do = nn.Dropout(0.1)

        # lightning - defining the loss
        self.loss = nn.CrossEntropyLoss()

And the forward pass as:

# pytorch - Forward pass is the same as for a normal torch.nn module
    def forward(self, x):
        h1 = nn.functional.relu(self.l1(x))
        h2 = nn.functional.relu(self.l2(h1))
        do = self.do(h2 + h1)
        logits = self.l3(do)
        return logits

Just like in the Masterclass Video
The import is:
from pytorch_lightning.metrics.functional import accuracy
any ideas how to fix this, any help would be greatly appreciated!

Cheers
Nico

[rchen19]

@NicoMandel As the error message suggested, you need apply either sigmoid, for binary classification, or softmax for multiclass classification, to the logits you get from the model output, before you pass it to the metric.

[NicoMandel]

Hi, thank you for that.
Would I do that by appending another layer to the model, or do I do it after the loss? So is it added to the __init__ as:

self.l1 = nn.Linear(28 * 28, 64)
        self.l2 = nn.Linear(64, 64)
        self.l3 = nn.Linear(64, 10)
        self.do = nn.Dropout(0.1)
        self.softm = nn.Softmax()

or in the traiing_step as

J = self.loss(logits, y)
nor = nn.softmax(x)
acc = accuracy(nor)

I thought softmax and Cross-entropy should not be used together? Nielsen

[rchen19]

@NicoMandel depending on the loss you use, I see you are using CrossEntropyLoss, not NNLLoss, so your loss accepts logits, then you need to apply softmax right before your accuracy metric.

[daspk04]

Hello @NicoMandel ,

The only part you need to change is accuracy.

    # 2 computing the loss function
    J = self.loss(logits, y)

    # lightning 4 - also getting the accuracy and showing it in the pbar
    acc = accuracy(F.softmax(logits, dim=1), y)          #Apply softmax to logits here
    pbar = {'train_acc': acc}