different results from paper

[ywyga]

Hi, and thank you for this valuable survey.

I noticed that in the preprint your results for Graph WaveNet are substantially worse than what was reported in the original paper.
For reference, Graph WaveNet MAE results for 15, 30 and 60 min are: 2.69, 3.07, 3.53
While in the preprint the results are: 3.204, 3.922, 4.848
Perhaps i missed something but the hyperparameters and data splits seem similar in both cases.
How do you explain this difference?

[deepkashiwa20]

Thanks for your attention. The quick answer is, GW-Net uses masked_mae_loss while we just use the original mae loss.
Because METR-LA dataset has lots of null values (also regarded as 0), masked_mae_loss used in their original repository (GW-Net) makes a great improvement. However, in order to evaluate the pure model performances, we take mae as the loss in our project. You can get a very close result to the original paper by using masked_mae_loss. We have validated this.

[ywyga]

Thanks for the reply. I see it now. But if you don't mind me asking, why not use the masked loss for the benchmark? After all the evaluation uses masked MAE and apparently the masked loss reaches better result than the non-masked (at least in gwnet)

[deepkashiwa20]

Yes, you are right, on METR-LA, apparently mask_mae is better, but on PEMS-BAY, the difference is not so big. This all depends on the dataset property. We adopt the most naive MAE loss as the unified loss function to verify the "pure" performances of the models, by excluding other factors (e.g., loss function, extra data source). This is our originial idea. But, of course, on METR-LA, we should use masked MAE as the unified loss, definitely. Hope we could update/improve this point in the future.

[zezhishao]

The masking metrics may be more reasonable.
This is because if we compute loss on outliers (e.g., zero values in METR-LA), it is equivalent to forcing the model to fit these outliers, which makes no sense, and more importantly, it leads to worse prediction accuracy on the other normal values.