# Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

**Authors:** Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov  
**arXiv:** [1901.02860](https://arxiv.org/abs/1901.02860)  
**Date:** 2019-01-09  
**Summary page:** https://www.summarizepaper.com/en/paper/1901.02860v1/

## Key points

- Language modeling has seen advancements in recent years
- Traditional RNNs face challenges in optimizing gradient vanishing and explosion
- Transformer networks have shown promise in learning longer-term dependency but are constrained by a fixed-length context
- The authors propose a new neural architecture called Transformer-XL
- Transformer-XL incorporates a segment-level recurrence mechanism and a novel positional encoding scheme
- Transformer-XL captures longer-term dependency and resolves the problem of context fragmentation
- Transformer-XL learns dependencies that are approximately 80% longer than RNNs and 450% longer than vanilla Transformers
- Transformer-XL achieves better performance on both short and long sequences while being faster during evaluation
- The authors improve upon the state-of-the-art results for bit-per-character (bpc) and perplexity metrics on benchmark datasets such as enwiki8, text8, WikiText-103, One Billion Word, and Penn Treebank (without finetuning)
- Code, pretrained models, and hyperparameters are provided in both Tensorflow and PyTorch for reproducibility

## Summary

The field of language modeling has seen significant advancements in recent years, with researchers devising novel architectures and optimization algorithms to improve context encoding and capture long-term dependency. However, traditional recurrent neural networks (RNNs) face challenges in optimizing gradient vanishing and explosion, limiting their ability to model long-range dependencies effectively. In contrast, Transformer networks have shown promise in learning longer-term dependency but are constrained by a fixed-length context. To address these limitations, the authors propose a new neural architecture called Transformer-XL. This architecture incorporates a segment-level recurrence mechanism and a novel positional encoding scheme that enables the Transformer network to learn dependencies beyond a fixed length without disrupting temporal coherence. By doing so, Transformer-XL not only captures longer-term dependency but also resolves the problem of context fragmentation. The authors evaluate Transformer-XL's performance on various language modeling tasks and compare it with RNNs and vanilla Transformers. They find that Transformer-XL learns dependencies that are approximately 80% longer than RNNs and 450% longer than vanilla Transformers. Additionally, it achieves better performance on both short and long sequences while being significantly faster during evaluation. Furthermore, the authors improve upon the state-of-the-art results for bit-per-character (bpc) and perplexity metrics on several benchmark datasets such as enwiki8, text8, WikiText-103, One Billion Word, and Penn Treebank (without finetuning). They provide code, pretrained models, and hyperparameters in both Tensorflow and PyTorch for reproducibility. In summary, this paper introduces Transformer-XL as a solution to learning longer-term dependency in language modeling. The proposed architecture outperforms RNNs and vanilla Transformers in capturing dependencies over extended contexts while maintaining temporal coherence. The improved results on benchmark datasets demonstrate the effectiveness of Transformer-XL in addressing the limitations of existing models.