Is this a bug? - unnecessary CPU/GPU copying in supporters.py just for aggregating loss

[isvogor-foi]

I've noticed that while training, this chunk here is eating a lot of time.

https://github.com/PyTorchLightning/pytorch-lightning/blob/0b682b807a76b732492a3d43e3208a80c80b6c2f/pytorch_lightning/trainer/supporters.py#L76-L80

I'm wondering whether this is a bug, and why we don't have it like:

 self.memory = torch.zeros(self.window_length, *x.shape, device=x.device)

It works, and it's much, faster.

cc @Borda @akihironitta

[rohitgr7]

this will probably be removed in some time: #9372

[carmocca]

This is a very old piece of code. I guess the choice was to prefer less GPU memory usage over speed

[isvogor-foi]

Well, I'm not sure how impactful this is on the memory, but it's an easy fix. Should I submit a PR?
I've made some measurements... So in the advance function I've measured how much does it take to perform a training step.. And then I've summed it up through all epochs, for a simple MNIST example.

Never mind the other bars in the plot, but focus on the red highlighted one. The total time to run is around 105 S, out of this time the green bar (10 S) is the training, the remaining time, red, is performing the loss update.

Just adding the x.device() fixes this. I see no reason why this would not be fixed...

[carmocca]

Yes, go ahead!