# Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

**Authors:** Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, Dietrich Klakow  
**arXiv:** [2010.03179](https://arxiv.org/abs/2010.03179)  
**Date:** 2020-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.03179v1/

## Key points

- Authors explore capabilities of multilingual transformer models like mBERT and XLM-RoBERTa in NLP tasks across languages
- Challenge of transferring results from high-resource to low-resource languages highlighted
- Focus on African languages (Hausa, isiXhosa, Yorùbá) for Named Entity Recognition and topic classification tasks
- Transfer learning and distant supervision techniques enable comparable performance to baselines with minimal labeled data
- Certain scenarios identified where performance parity may not hold true
- Insights into challenges and opportunities of low-resource learning in NLP tasks specifically for African languages

## Summary

In their paper "Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages," authors Michael A. Hedderich, David Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, and Dietrich Klakow explore the capabilities of multilingual transformer models such as mBERT and XLM-RoBERTa. These models have shown significant advancements in various natural language processing (NLP) tasks across a wide range of languages. However, recent research has highlighted the challenge of effectively transferring results from high-resource languages to low-resource scenarios. To address this issue, the authors focus on three African languages - Hausa, isiXhosa, and Yor\`ub\'a - and investigate performance trends based on varying levels of available resources for Named Entity Recognition (NER) and topic classification tasks. Through their study, they demonstrate that by leveraging transfer learning or distant supervision techniques, these multilingual transformer models can achieve comparable performance to baselines with significantly more labeled training data using as few as 10 or 100 labeled sentences. While these results are promising, the authors also identify certain scenarios where this level of performance parity does not hold true. Their discussions and additional experiments shed light on key assumptions such as time constraints and hardware limitations that pose challenges but also present opportunities in the realm of low-resource learning. Accepted at EMNLP'20, this research contributes valuable insights into the effectiveness of transfer learning and distant supervision strategies for enhancing NLP tasks in low-resource settings specifically focusing on African languages. The findings underscore both the potential and limitations of utilizing multilingual transformer models in scenarios with limited labeled data availability.