In recent years, there has been a surge of interest in using Transformer-based solutions for time series forecasting (TSF), particularly for the challenging long-term TSF problem. The Transformer architecture relies on self-attention mechanisms to extract semantic correlations between paired elements in a long sequence, which is permutation-invariant and anti-ordering to some extent. However, in time series modeling, we are interested in extracting temporal relations among an ordering set of continuous points. This raises the question of whether Transformer-based techniques are suitable for long-term time series forecasting despite their demonstrated performance improvements. To investigate this issue further, a study was conducted comparing Transformer-based models with an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting. DLinear decomposes the time series into a trend and remainder series and employs two one-layer linear networks to model these two series for the forecasting task. Surprisingly, DLinear outperformed existing complex Transformer-based models in most cases by a large margin. The study also compared the performance of FEDformer and Autoformer against DLinear under a longer look-back window size of 336. The results showed that DLinear largely surpassed these state-of-the-art Transformer-based methods on datasets such as Exchange rate, Traffic, Electricity, Weather, ETTm1 and ETTh1. The study concludes that the relatively higher long-term forecasting accuracy of Transformer-based TSF solutions shown in existing works has little to do with the temporal relation extraction capabilities of the Transformer architecture. Instead, it is mainly due to the nonautoregressive DMS forecasting strategy used in them. This finding raises questions about the validity of using Transformers for other time series analysis tasks such as anomaly detection. Overall, this study highlights the importance of questioning established assumptions regarding model suitability for specific tasks and encourages researchers to explore simpler architectures before resorting to more complex ones. Section 6 concludes this paper by discussing potential future research directions.
A study compared two types of models for predicting things that happen over time. One type is called a Transformer, which has become popular recently. The other type is called DLinear and is very simple. The study found that DLinear was better at predicting in most cases, even though Transformers are more complex. This means we should question whether Transformers are always the best choice for predicting things that happen over time.
Definitions
- Time series forecasting (TSF): Predicting what will happen over time based on past data.
- Transformers: A type of model that uses self-attention mechanisms to find connections between different parts of a long sequence.
- Self-attention mechanisms: A way for models to focus on important parts of a sequence by weighting them differently.
- Embarrassingly simple architecture: A very basic model with few parameters or features.
- Direct multi-step (DMS) forecasting: Predicting multiple steps ahead without using previous predictions as input.
Exploring the Use of Transformer-Based Solutions for Time Series Forecasting
In recent years, there has been a surge of interest in using Transformer-based solutions for time series forecasting (TSF). This is due to their demonstrated performance improvements compared to traditional methods. However, it is important to question whether these models are suitable for long-term TSF tasks given that they rely on self-attention mechanisms which are permutation-invariant and anti-ordering. To investigate this issue further, a study was conducted comparing Transformer-based models with an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting. The results showed that DLinear largely surpassed state-of-the art Transformer based methods such as FEDFormer and Autoformer on datasets such as Exchange rate, Traffic, Electricity, Weather, ETTm1 and ETTh1.
Background: What is Time Series Forecasting?
Time series forecasting (TSF) is the process of predicting future values from past data points. It can be used in various fields such as finance or weather prediction. Traditional approaches to TSF involve using linear regression or autoregressive integrated moving average (ARIMA) models which have limited accuracy when dealing with long term predictions.
The Rise of Transformers for Time Series Forecasting
Recently there has been an increase in the use of transformer architectures for time series forecasting due to their ability to extract semantic correlations between paired elements in a long sequence which makes them permutation invariant and anti ordering to some extent. These properties make them well suited for tasks like language translation where order does not matter but understanding meaning does. However when it comes to time series modeling we are interested in extracting temporal relations among an ordering set of continuous points so it raises the question if transformers are still suitable despite their demonstrated performance improvements?
Comparing Transformers with DLinear
To answer this question a study was conducted comparing transformer based models with an embarrassingly simple architecture called DLinear which conducts direct multi step (DMS) forecasting by decomposing the time series into two parts - trend and remainder - then employing two one layer linear networks to model these two components respectively . Surprisingly DLinear outperformed existing complex transformer based models by a large margin even under longer look back window sizes up 336 . This finding suggests that the higher long term forecast accuracy seen from transformers may have little do with its temporal relation extraction capabilities but rather its nonautoregressive DMS strategy employed within them .
Implications & Future Directions
This study highlights how important it is to question established assumptions regarding model suitability for specific tasks before resorting more complex ones . It also raises questions about validity of using transformers for other time series analysis tasks such as anomaly detection . In terms of future research directions , potential areas include exploring ways improve upon existing nonautoregressive strategies or investigating alternative architectures better suited towards capturing temporal relations within time series data sets .
Recently, there has been a surge of Transformer-based solutions for the time series forecasting (TSF) task, especially for the challenging long-term TSF problem. Transformer architecture relies on self-attention mechanisms to effectively extract the semantic correlations between paired elements in a long sequence, which is permutation-invariant and anti-ordering to some extent. However, in time series modeling, we are to extract the temporal relations among an ordering set of continuous points. Consequently, whether Transformer-based techniques are the right solutions for long-term time series forecasting is an interesting problem to investigate, despite the performance improvements shown in these studies. In this work, we question the validity of Transformer-based TSF solutions. In their experiments, the compared (non-Transformer) baselines are mainly autoregressive forecasting solutions, which usually have a poor long-term prediction capability due to inevitable error accumulation effects. In contrast, we use an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting for comparison. DLinear decomposes the time series into a trend and a remainder series and employs two one-layer linear networks to model these two series for the forecasting task. Surprisingly, it outperforms existing complex Transformer-based models in most cases by a large margin. Therefore, we conclude that the relatively higher long-term forecasting accuracy of Transformer-based TSF solutions shown in existing works has little to do with the temporal relation extraction capabilities of the Transformer architecture. Instead, it is mainly due to the non-autoregressive DMS forecasting strategy used in them. We hope this study also advocates revisiting the validity of Transformer-based solutions for other time series analysis tasks (e.g., anomaly detection) in the future.
