# iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

**Authors:** Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, Mingsheng Long  
**arXiv:** [2310.06625](https://arxiv.org/abs/2310.06625)  
**Date:** 2023-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.06625v4/

## Key points

- Authors address the current trend of linear forecasting models challenging the need for architectural modifications in Transformer-based forecasters
- Traditional Transformers face limitations when forecasting series with larger lookback windows due to performance degradation and computational complexity
- iTransformer applies attention and feed-forward networks on inverted dimensions to capture multivariate correlations and learn nonlinear representations
- iTransformer achieves state-of-the-art performance on challenging real-world datasets
- The proposed model enhances the Transformer family by improving performance, generalization across different variates, and enabling better utilization of arbitrary lookback windows

## Summary

In their paper titled "iTransformer: Inverted Transformers Are Effective for Time Series Forecasting," authors Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long address the current trend of linear forecasting models that challenge the need for architectural modifications in Transformer-based forecasters. These forecasters typically use Transformers to capture global dependencies across temporal tokens in time series data, where each token consists of multiple variates from the same timestamp. However, traditional Transformers face limitations when forecasting series with larger lookback windows due to performance degradation and computational complexity. The authors highlight issues with the existing approach where the embedding for each temporal token combines multiple variates representing potential delayed events and distinct physical measurements. This fusion may hinder the model's ability to learn variate-centric representations effectively, leading to meaningless attention maps. To overcome these challenges without modifying the core components of the Transformer architecture, they introduce iTransformer. iTransformer applies attention and feed-forward networks on inverted dimensions. Specifically, individual time points within a series are embedded into variate tokens used by the attention mechanism to capture multivariate correlations. Simultaneously, the feed-forward network processes each variate token to learn nonlinear representations. The proposed iTransformer model achieves state-of-the-art performance on challenging real-world datasets. This novel approach enhances the Transformer family by improving performance, generalization across different variates, and enabling better utilization of arbitrary lookback windows. iTransformer emerges as a promising alternative as a fundamental backbone for time series forecasting tasks. The code for implementing iTransformer is available at https://github.com/thuml/iTransformer. Overall,this research sheds light on repurposing Transformer components effectively for time series forecasting without compromising on performance or scalability.