# Consistency Regularization for Cross-Lingual Fine-Tuning

**Authors:** Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei  
**arXiv:** [2106.08226](https://arxiv.org/abs/2106.08226)  
**Date:** 2021-06-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.08226v1/

## Key points

- The paper proposes a method to enhance cross-lingual fine-tuning using consistency regularization.
- Example consistency regularization is introduced to penalize prediction sensitivity in response to four types of data augmentations: subword sampling, Gaussian noise, code-switch substitution, and machine translation.
- By incorporating these augmentations into the training process and encouraging consistent predictions across different augmented versions of the same input example, the model becomes more robust and better generalizes across languages.
- The effectiveness of the proposed approach is evaluated on the XTREME benchmark dataset across various tasks such as text classification, question answering, and sequence labeling.
- Experimental results demonstrate that consistency regularization significantly improves cross-lingual fine-tuning performance compared to existing methods.

## Summary

The paper titled "Consistency Regularization for Cross-Lingual Fine-Tuning" by Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song and Furu Wei proposes a method to enhance cross-lingual fine-tuning using consistency regularization. The objective is to leverage pre-trained cross-lingual language models to transfer task-specific supervision from one language to others effectively. To achieve this goal, the authors introduce example consistency regularization as a means to penalize prediction sensitivity in response to four types of data augmentations: subword sampling, Gaussian noise, code-switch substitution and machine translation. By incorporating these augmentations into the training process and encouraging consistent predictions across different augmented versions of the same input example, the model becomes more robust and better generalizes across languages. The effectiveness of the proposed approach is evaluated on the XTREME benchmark dataset across various tasks such as text classification, question answering and sequence labeling. Experimental results demonstrate that the introduced consistency regularization significantly improves cross-lingual fine-tuning performance compared to existing methods. In summary, this work presents a novel technique for enhancing cross-lingual fine-tuning through consistency regularization. By leveraging data augmentations and encouraging consistent predictions across different versions of augmented examples during training, the proposed method achieves notable improvements in cross-lingual performance across multiple tasks.