# Unsupervised Cross-lingual Representation Learning at Scale

**Authors:** Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov  
**arXiv:** [1911.02116](https://arxiv.org/abs/1911.02116)  
**Date:** 2019-11-05  
**Summary page:** https://www.summarizepaper.com/en/paper/1911.02116v1/

## Key points

- Study demonstrates effectiveness of pretraining multilingual language models on a large scale
- Transformer-based masked language model, XLM-R, trained on over two terabytes of CommonCrawl data from 100 languages
- XLM-R outperforms mBERT on cross-lingual benchmarks
- Average accuracy improvement of 13.8% on XNLI task, F1 score improvements of 12.3% on MLQA and 2.1% on NER
- Performs exceptionally well on low-resource languages (11.8% accuracy improvement for Swahili, 9.2% improvement for Urdu)
- Detailed empirical evaluation explores trade-offs between positive transfer and capacity dilution, as well as performance of high and low resource languages at scale
- XLM-R highly competitive with strong monolingual models on GLUE and XNLI benchmarks
- Code, data, and models for XLM-R to be made publicly available for further research advancement
- Research highlights significant performance gains in pretraining multilingual language models at scale and provides insights into optimizing such models for cross-lingual transfer tasks across diverse languages

## Summary

This paper, titled "Unsupervised Cross-lingual Representation Learning at Scale," presents the findings of a study that demonstrates the effectiveness of pretraining multilingual language models on a large scale. The researchers trained a Transformer-based masked language model, called XLM-R, on a dataset consisting of more than two terabytes of filtered CommonCrawl data from one hundred languages. The results show that XLM-R outperforms multilingual BERT (mBERT) on various cross-lingual benchmarks. It achieves an average accuracy improvement of 13.8% on the XNLI task, an average F1 score improvement of 12.3% on MLQA and an average F1 score improvement of 2.1% on NER. Notably, XLM-R performs exceptionally well on low-resource languages with an 11.8% accuracy improvement for Swahili and a 9.2% improvement for Urdu compared to the previous XLM model. The paper also provides a detailed empirical evaluation of the key factors contributing to these performance gains by exploring trade-offs between positive transfer and capacity dilution as well as examining how high and low resource languages perform at scale. This study demonstrates that multilingual modeling can be achieved without sacrificing per-language performance; indeed, XLM-R proves to be highly competitive with strong monolingual models on the GLUE and XNLI benchmarks. The authors plan to make the code, data and models for XLM-R publicly available so other researchers can benefit from their findings and further advance cross-lingual representation learning at scale. Overall, this research highlights the significant performance gains that can be achieved by pretraining multilingual language models at scale and provides valuable insights into optimizing such models for cross-lingual transfer tasks across diverse languages.