RevIN

[gdevos010]

I'm not sure if you are looking for additional papers, but RevIN has been found to improve forecasting among many different kinds of models. It's a pretty simple addition where you normalize before the forward pass and then denormalize afterwards.

Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift
repo

[lucidrains]

hey Greg

without reading the paper you linked, is the technique orthogonal to the paper in the repo? also, i'm only putting in more work if someone tells me the inverted transformer actually works or not

[lucidrains]

@gdevos010 i see you are working with medical time series. tell you what, you run inverted transformer on this, show me that it is at least competitive, and i'll invest the time to read the RevIN paper and figure out how to integrate it, if ideas do not conflict, and if it passes my smell test

[lucidrains]

we can also move this to discussions, as it is not an issue with the iTransformer paper

[lucidrains]

@gdevos010 hey Greg, the authors released their official code this morning, so likely the paper is going to pan out

i can do a bit of improvisation and get this idea into the repository, just to see if we can push time series sota even further

[lucidrains]

ok added

[lucidrains]

import torch
from iTransformer import RevIN

x = torch.randn(1, 512, 1024)
rev_in = RevIN(512)
out, reverse_fn = rev_in(x)
y = reverse_fn(out)

assert torch.allclose(x, y)

[lucidrains]

@gdevos010 do let us know if it works in this context, or not

[gdevos010]

@lucidrains Awesome! Thanks for adding. The implementation looks right to me and Im hoping to test out the model this weekend

[lucidrains]

@gdevos010 how did it go?

[gdevos010]

@lucidrains I added the model to darts framework locally and compared it against some of the other models. It performed worse than other models, but I will keep looking into it and will try tuning the hyperparameters.

input len 250, output len 36

input len 250, output len 100

[lucidrains]

@gdevos010 ahh ok, so RevIN has no effect? maybe i should just remove it

[gdevos010]

@lucidrains I was not comparing RevIN on/off. I only has time for few experiments yesterday and was just testing the model. I will test that today

[gdevos010]

Yes, these are multivariate time series

[lucidrains]

@gdevos010 nice, thank you for sharing this!

[lucidrains]

@gdevos010 want to give the 2d version a try as well? while you are at it?

[gdevos010]

@lucidrains I spent some time today trying to get the RevIN working. There's an issue with the tensor shape that I still need to figure out. I believe this has to do with my integration into darts instead of your implementation.

I also got the 2D version up and running. it is much slower to train. Here is one of the example outputs and I will run it overnight to get more data.

[gdevos010]

@lucidrains I tested it with 5 more datasets. This hydro energy dataset is hard for a lot of models to learn but itransformer does amazing on it.

[lucidrains]

this is great news! thank you so much Greg!

[lucidrains]

for the hydro experiments, was RevIn turned on or off?

[kashif]

@gdevos010 so i have a probabilistic variant of iTransform in this PR: https://github.com/awslabs/gluonts/blob/97577be22607ff5c2b5a992c820fd756bf3e2061/examples/iTransformer.ipynb

recall that the RevIN is instance-normalization where the mu and sigma are then used to rescale back the emission on the output side... this technique is related to the original one I believe first done in the DeepAR paper and I have implemented it in the PR where the distribution or its parameters are rescaled back. Also like the Non-stationary transformer I pass the mu and sigma as static covariates (which iTransformer works with unlike dynamic ones) and this technique is also implemented in my PR and I believe is an indirect way (unlike the explicit way in Non-stationary transformer paper) check it out!