# When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer

**Authors:** Ameet Deshpande, Partha Talukdar, Karthik Narasimhan  
**arXiv:** [2110.14782](https://arxiv.org/abs/2110.14782)  
**Date:** 2021-10-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.14782v3/

## Key points

- Recent work on multilingual language models has demonstrated their ability to transfer knowledge across languages in zero-shot settings.
- Lack of consensus in the community regarding shared properties between languages that enable such transfer.
- Authors conducted a large-scale empirical study to investigate effects of different linguistic properties on zero-shot transfer.
- Measured transfer performance between four diverse natural languages and modified counterparts.
- Absence of sub-word overlap significantly affects zero-shot transfer when languages have different word orders.
- Strong correlation between transfer performance and word embedding alignment between languages (correlation coefficient of 0.94 for NLI task).
- Argue for focus on explicitly improving word embedding alignment between languages in multilingual models.
- Need for understanding specific linguistic properties that facilitate cross-lingual zero-shot transfer.
- Sub-word overlap and word embedding alignment play crucial roles in enabling successful transfer between languages.
- Insights can inform future research efforts aimed at enhancing multilingual models' performance and generalization capabilities.

## Summary

Recent work on multilingual language models has demonstrated their ability to transfer knowledge across languages in zero-shot settings. However, there is a lack of consensus in the community regarding the shared properties between languages that enable such transfer. This is because natural languages differ in various linguistic aspects, making it difficult to isolate the effects of specific properties. To address this issue, the authors conducted a large-scale empirical study to investigate the effects of different linguistic properties on zero-shot transfer. They measured transfer performance between four diverse natural languages and their counterparts constructed by modifying aspects such as script, word order, and syntax. The experiments revealed several interesting findings. Firstly, they found that the absence of sub-word overlap significantly affects zero-shot transfer when languages have different word orders. Additionally, they observed a strong correlation between transfer performance and word embedding alignment between languages. For example, on the task of Natural Language Inference (NLI), there was a correlation coefficient of 0.94. Based on these results, the authors argue for a focus on explicitly improving word embedding alignment between languages in multilingual models rather than relying on its implicit emergence. In summary, this paper highlights the need for understanding the specific linguistic properties that facilitate cross-lingual zero-shot transfer. The findings suggest that sub-word overlap and word embedding alignment play crucial roles in enabling successful transfer between languages. These insights can inform future research efforts aimed at enhancing multilingual models' performance and generalization capabilities across diverse language pairs.