# XLNet: Generalized Autoregressive Pretraining for Language Understanding

**Authors:** Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le  
**arXiv:** [1906.08237](https://arxiv.org/abs/1906.08237)  
**Date:** 2019-06-19  
**Summary page:** https://www.summarizepaper.com/en/paper/1906.08237v1/

## Key points

- XLNet is a generalized autoregressive pretraining method that improves upon BERT
- XLNet enables bidirectional context modeling and uses an autoregressive formulation
- XLNet integrates ideas from Transformer-XL, a state-of-the-art autoregressive model
- XLNet outperforms BERT on 20 tasks, achieving state-of-the-art results on 18 tasks
- XLNet uses various datasets for pretraining, including BooksCorpus, English Wikipedia, Giga5, ClueWeb 2012-B, and Common Crawl
- The largest model (XLNet-Large) has similar architecture hyperparameters to BERT-Large and is trained on 512 TPU v3 chips for 500K steps
- Further training does not significantly improve downstream tasks after underfitting the data at the end of training
- An ablation study focuses on understanding the importance of different design choices in XLNet
- The study evaluates the effectiveness of permutation language modeling objective compared to denoising auto-encoding used by BERT
- It also examines the significance of using Transformer-XL as the backbone neural architecture with segment level recurrence
- Implementation details like span based prediction and next sentence prediction are considered in the ablation study
- Overall, XLNet proves to be a promising pretraining method that surpasses BERT's performance while incorporating advancements from Transformer-XL and addressing its limitations.

## Summary

XLNet is a generalized autoregressive pretraining method that addresses the limitations of BERT, a popular pretraining approach. While BERT achieves better performance than autoregressive language modeling, it neglects the dependency between masked positions and suffers from a pretrain-finetune discrepancy. XLNet overcomes these issues by enabling bidirectional context modeling and using an autoregressive formulation. It also integrates ideas from Transformer-XL, a state-of-the-art autoregressive model. In terms of experiments, XLNet outperforms BERT on 20 tasks, including question answering, natural language inference, sentiment analysis and document ranking. It achieves state-of-the-art results on 18 tasks and demonstrates the usefulness of recent advancements in language modeling research. For pretraining, XLNet uses various datasets such as BooksCorpus, English Wikipedia, Giga5, ClueWeb 2012-B and Common Crawl. The largest model (XLNet-Large) has similar architecture hyperparameters to BERT-Large and is trained on 512 TPU v3 chips for 500K steps. Despite underfitting the data at the end of training further training does not improve downstream tasks significantly. The ablation study conducted on four datasets focuses on understanding the importance of different design choices in XLNet. This includes evaluating the effectiveness of permutation language modeling objective compared to denoising auto-encoding used by BERT; the significance of using Transformer-XL as the backbone neural architecture with segment level recurrence; and bidirectional input pipeline implementation details like span based prediction and next sentence prediction. Overall, XLNet proves to be a promising pretraining method that surpasses BERT's performance across various tasks while incorporating advancements from Transformer-XL and addressing its limitations.