# Cross-lingual Language Model Pretraining

**Authors:** Guillaume Lample, Alexis Conneau  
**arXiv:** [1901.07291](https://arxiv.org/abs/1901.07291)  
**Date:** 2019-01-22  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1901.07291v1/

## Key points

- Study explores cross-lingual pretraining using generative models for English natural language understanding
- Two methods proposed: unsupervised approach with monolingual data, supervised approach with parallel data and new cross-lingual language model objective
- Achieved state-of-the-art results in cross-lingual classification, unsupervised machine translation, and supervised machine translation tasks
- Outperformed existing approaches in cross-lingual classification with 4.9% accuracy gain on XNLI
- Significant improvement in unsupervised machine translation with BLEU score of 34.3 on WMT'16 German–English (9 BLEU points higher than previous best)
- Set new state-of-the-art benchmark in supervised machine translation with BLEU score of 38.5 on WMT'16 Romanian–English (4 BLEU points higher than previous best)
- Code and pretrained models will be made publicly available for further research and development
- Study contributes to advancing natural language understanding across multiple languages

## Summary

The study titled "Cross-lingual Language Model Pretraining" by Guillaume Lample and Alexis Conneau explores the effectiveness of cross-lingual pretraining using generative models for English natural language understanding. The authors propose two methods to learn cross-lingual language models (XLMs): an unsupervised approach that relies solely on monolingual data, and a supervised approach that utilizes parallel data with a new cross-lingual language model objective. The research demonstrates the superiority of their approach by achieving state-of-the-art results in various tasks such as cross-lingual classification, unsupervised machine translation, and supervised machine translation. In terms of cross-lingual classification, their method outperforms existing approaches with an absolute gain of 4.9% accuracy on XNLI. For unsupervised machine translation, the authors achieve significant improvement over previous state-of-the art results by obtaining a BLEU score of 34.3 on WMT'16 German–English, surpassing the previous best performance by more than 9 BLEU points. In supervised machine translation, they set a new state–of–the–art benchmark with a BLEU score of 38.5 on WMT'16 Romanian–English, surpassing the previous best approach by more than 4 BLEU points. The authors plan to make their code and pretrained models publicly available to provide valuable resources for further research and development in this area. Overall, their study highlights the effectiveness of cross–lingual pretraining using generative models and contributes significantly to advancing natural language understanding across multiple languages.