SamFormer multivariate Estimator

[kashif]

Add samformer model https://arxiv.org/abs/2402.10198

cc @romilbert

[kashif]

@romilbert this implementation differs, I believe, from yours in that I am using the mu and sigma from the instance normalization as inputs to the model as well. This serves the purpose of RevIN + non-stationary transformer and was first done in the DeepAR paper, which as far as I remember the RevIN or the SAMFromer paper does not mention....

Apart from that the output head is an independent distribution head instead of a point forecasting head...

[romilbert]

Hi @kashif, thanks for your interest in SAMformer. I appreciate your feedback. Incorporating mu and sigma from RevIN as additional inputs to the model is an interesting idea (similar to the DeepAR paper as you said, which we did reference in SAMformer paper). However, I have tried incorporating them into the transformer in different ways, and it did not improve our current results.

[kashif]

@romilbert that is because, I think, you are using globally standardized data while here the data continues to be in the original magnitude, and thus in your setup the use of the summary statistics do not help as they are more or less 0 and 1...