# Make Transformer Great Again for Time Series Forecasting: Channel Aligned Robust Dual Transformer

**Authors:** Wang Xue, Tian Zhou, QingSong Wen, Jinyang Gao, Bolin Ding, Rong Jin  
**arXiv:** [2305.12095](https://arxiv.org/abs/2305.12095)  
**Date:** 2023-05-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.12095v1/

## Key points

- Recent studies have shown the effectiveness of deep learning methods for time series forecasting, including Transformer and MLP.
- Transformer is observed to be less effective than MLP in this task.
- The paper proposes a special Transformer called CARD to address the limitations of Transformer in time series forecasting.
- CARD incorporates a dual Transformer structure that captures both temporal correlations among signals and dynamical dependence among multiple variables over time.
- A robust loss function is introduced to mitigate potential overfitting issues by considering prediction uncertainties.
- CARD outperforms state-of-the-art models, including Transformer and MLP-based models, in various long-term and short-term forecasting datasets.
- Section 2 provides a summary of related works in the field of time series forecasting using Transformers, including innovative designs like convolutional self-attention layers or hierarchical attention mechanisms.
- Section 3 presents the detailed model architecture of CARD.
- Section 4 describes the design of its robust loss function with a theoretical explanation based on maximum likelihood estimation.
- Section 5 presents results from numerical experiments conducted on long-term and short-term time series forecasting benchmarks using different models, where CARD consistently outperforms other models across all prediction horizons.

## Summary

Recent studies have shown the effectiveness of deep learning methods, such as Transformer and MLP, for time series forecasting. However, it has been observed that Transformer is less effective than MLP in this task. In this paper, we propose a special Transformer called channel-aligned robust dual Transformer (CARD) to address the limitations of Transformer in time series forecasting. CARD incorporates a dual Transformer structure that captures both temporal correlations among signals and dynamical dependence among multiple variables over time. Additionally, we introduce a robust loss function that considers prediction uncertainties to mitigate potential overfitting issues. We evaluate CARD on various long-term and short-term forecasting datasets and compare it with state-of-the-art methods, including Transformer and MLP-based models. Our results demonstrate that CARD outperforms these models significantly. The remainder of the paper is organized as follows: Section 2 provides a summary of related works in the field of time series forecasting using Transformers. This includes LogTrans, Informer, Autoformer, FEDformer, Pyraformer, PatchTST and Crossformer which incorporate innovative designs like convolutional self-attention layers or hierarchical attention mechanisms to capture dependencies in time series data effectively. Section 2 also discusses the use of RNNs, MLPs and CNNs for time series forecasting which have been widely used in the past but may not fully exploit the potential of deep learning methods like Transformers. Section 3 presents the detailed model architecture of CARD while Section 4 describes the design of its robust loss function with a theoretical explanation based on maximum likelihood estimation. In Section 5 we present results from numerical experiments conducted on long-term and short-term time series forecasting benchmarks using different models including CARD along with FilM ETSFormer Statonary FEDFormer and AutoFormer. The evaluation metrics include mean squared error (MSE) and mean absolute error (MAE) for various prediction horizons where CARD consistently outperforms other models across all prediction horizons demonstrating its power for time series forecasting surpassing existing state-of-the art methods.