# Language Identification for Austronesian Languages

**Authors:** Jonathan Dunn, Wikke Nijhof  
**arXiv:** [2206.04327](https://arxiv.org/abs/2206.04327)  
**Date:** 2022-06-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.04327v1/

## Key points

- Language identification models for low- and under-resourced languages in the Pacific region, particularly Austronesian languages
- Goal is to develop accurate language identification systems as part of building language resources
- Evaluation set includes 29 Austronesian languages and 171 non-Austronesian languages from eight data sources
- Skip-gram embeddings outperform other methods significantly in language identification
- Minimal decrease in accuracy when expanding the inventory of non-Austronesian languages
- Adapting language identification models for code-switching detection with high accuracy across all 29 languages studied
- Performance and stability of models after compression evaluated
- Prioritizing diversity in data collection rather than reducing sample size
- Detailed overview of approach, including selection of non-Austronesian languages based on genetic classifications and previous work on digital language mapping
- Comparison of inventory of 200 languages with popular language identification packages such as Google's CLD32 and langid.py3
- Research contributes to improving language identification models for low-resource Pacific languages by demonstrating effectiveness of skip-gram embeddings and exploring impact of increasing non-Austronesian language inventories

## Summary

This paper focuses on language identification models for low- and under-resourced languages in the Pacific region, particularly Austronesian languages that were previously unavailable. The goal is to develop accurate language identification systems as part of building language resources. The study combines 29 Austronesian languages with 171 non-Austronesian languages from eight data sources to create an evaluation set. After evaluating six different approaches, the researchers find that a classifier based on skip-gram embeddings outperforms other methods significantly. To further investigate the impact of increasing the inventory of non-Austronesian languages, the researchers systematically increase the number of languages in the model up to a total of 800. Surprisingly, they find that there is only a minimal decrease in accuracy when expanding the inventory, suggesting that including more non-Austronesian languages does not negatively affect predictions for Austronesian languages. The paper also explores adapting these language identification models for code-switching detection and achieves high accuracy across all 29 languages studied. Additionally, the performance and stability of models after compression are evaluated. In terms of data collection, the researchers prioritize maintaining diversity across various domains rather than reducing sample size. They argue that documents with fewer than 100 characters are less useful for corpus-building purposes. The paper provides a detailed overview of their approach, including information on how they selected non-Austronesian languages for their initial model based on genetic classifications and previous work on digital language mapping. They compare their inventory of 200 languages with popular language identification packages such as Google's CLD32 and langid.py3. Overall, this research contributes to improving language identification models for low-resource Pacific languages by demonstrating the effectiveness of skip-gram embeddings and exploring the impact of increasing non-Austronesian language inventories. The findings have implications for developing accurate code-switching detection systems as well.