Model does not seem to behave well with trending data

[juan-carvajal]

Hi! I saw your model and thought it was really cool! I tried to use it to predict some data, and it did seem to pickup the patterns really well, but it really struggled with trend data. For example:

I tried to train using this format:


But ended up with the following:

The model seems to undestimate the test data (kind of makes sense, because it was trained with a part of the data that had lower values).

Do you know any way to correct this?

Thanks by the way, you code was helpful and explained a lot of concepts.

[ly1112]

ied to train using this format:

Hi, sir. Have you solved this issue? Also has similar doubts, thanks.

[lkulowski]

Hi,

This issue is expected when modeling time series data with deep learning methods, and is not due to the code itself. You will need to detrend your data before training the LSTM encoder-decoder. There are different ways you can do this. One option would be to detrend by differencing. This involves constructing a new time series that contains the differences between consecutive values in the original time series. Another option would be to detrend by fitting a model. For your data, you could subtract out the linear trend. After detrending, make sure to standardize the new time series. The LSTM encoder-decoder will produce better results if the time series data has a consistent scale.

For information on detrending and standardizing the data, check out these articles:

https://machinelearningmastery.com/time-series-trends-in-python/ 
https://machinelearningmastery.com/normalize-standardize-time-series-data-python/

Good luck!

[ly1112]

Hi,

This issue is expected when modeling time series data with deep learning methods, and is not due to the code itself. You will need to detrend your data before training the LSTM encoder-decoder. There are different ways you can do this. One option would be to detrend by differencing. This involves constructing a new time series that contains the differences between consecutive values in the original time series. Another option would be to detrend by fitting a model. For your data, you could subtract out the linear trend. After detrending, make sure to standardize the new time series. The LSTM encoder-decoder will produce better results if the time series data has a consistent scale.

For information on detrending and standardizing the data, check out these articles:

https://machinelearningmastery.com/time-series-trends-in-python/ 
https://machinelearningmastery.com/normalize-standardize-time-series-data-python/

Good luck!

Hi Author, thanks so much for your clearification! Also I am trying to use your code to do reconstruction of the trained sequences rather than prediction part of it. So can you help me to check which paramaters or parts I need to change on your codes to achieve this？ Thanks so much!