# A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models

**Authors:** Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, jiwung Hyun, Sungho Park  
**arXiv:** [2306.02254](https://arxiv.org/abs/2306.02254)  
**Date:** 2023-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.02254v1/

## Key points

- Polyglot project aims to enhance non-English language performance of multilingual language models
- Researchers and developers often build monolingual models due to dissatisfaction with current multilingual models' non-English language capabilities
- Advanced multilingual language models developed for improved performance in non-English languages
- Introduction of Polyglot Korean models with a specific focus on Korean language
- Collaboration with TUNiB to collect 1.2TB of curated Korean data
- Prioritization of Korean models before venturing into multilingual models for performance comparisons and catering to specific needs of Korean companies and researchers
- Development of Polyglot-Ko model with three different sizes: 400M, 5.8B, and 12.8B parameters
- The 12.8 billion parameter model is the largest publicly available Korean language model suitable for commercial applications
- Assessment of zero-shot and few-shot performance using KOBEST benchmark shows competitive results across various datasets
- Open-source large-scale Korean language model that improves non-English language capabilities in multilingual language models

## Summary

Polyglot is a pioneering project aimed at enhancing the non-English language performance of multilingual language models. Despite the availability of various multilingual models, researchers and developers often resort to building monolingual models in their respective languages due to dissatisfaction with the current multilingual models' non-English language capabilities. To address this gap, the authors developed advanced multilingual language models that offer improved performance in non-English languages. In this paper, the authors introduce the Polyglot Korean models, which represent a specific focus rather than being multilingual in nature. In collaboration with TUNiB, their team collected 1.2TB of Korean data meticulously curated for their research journey. They made a deliberate decision to prioritize the development of Korean models before venturing into multilingual models. This choice was motivated by multiple factors: firstly, the Korean models facilitated performance comparisons with existing multilingual models; and finally, they catered to the specific needs of Korean companies and researchers. The authors present their work in developing the Polyglot Korean models, which propose some steps towards addressing the non-English language performance gap in multilingual language models. The Polyglot-Ko model represents one of their achievements, with three different sizes available: 400M, 5.8B, and 12.8B parameters. The 12.8 billion parameter model is particularly noteworthy as it is currently the largest publicly available Korean language model suitable for commercial applications. The authors assess the zero-shot and few-shot performance of their Polyglot-Ko models using the KOBEST benchmark and successfully demonstrate competitive results across various benchmark datasets. Overall, this paper presents an open-source large-scale Korean language model that contributes to improving non-English language capabilities in multilingual language models. It provides valuable resources for researchers and practitioners engaged in Korean natural language processing tasks and showcases advancements in addressing the non-English language performance gap.