[ENH] Truncated Backpropagation Through Time

[svnv-svsv-jm]

Is Truncated Backpropagation Through Time supported?

I've found no documentation or example about it.

[benHeid]

It seems that it is not supported. At least I haven't found it in training_step nor in the step method implemented. I think this could be a nice add on for autoregressive or recurrent networks.

For implementing it, I suppose it would require to add for the recurrent models a new step method or add a hook in the step method of the BaseModel for the calculation of the loss.

[svnv-svsv-jm]

PyTorch Lightning had this feature, then they removed it. We could take inspiration from their old code?

[benHeid]

I suppose the advise of PyTorch lightning is using manual optimization for that.

https://lightning.ai/docs/pytorch/stable/model/manual_optimization.html

Pull Requests are appreciated :)