# Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

**Authors:** Seungduk Kim, Seungtaek Choi, Myeongho Jeong  
**arXiv:** [2402.14714](https://arxiv.org/abs/2402.14714)  
**Date:** 2024-02-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.14714v1/

## Key points

- Introduction of \texttt{EEVE-Korean-v1.0}, a Korean adaptation of large language models
- Utilizes the Efficient and Effective Vocabulary Expansion (EEVE) method for parameter freezing and subword initialization
- EEVE-Korean-10.8B-v1.0 model excels in processing Korean language tasks while maintaining proficiency in English with just 2 billion tokens
- Project aims to expand vocabulary expansion methodology to additional languages for generalizability and effectiveness
- Focus on evaluating reasoning and generative capabilities through tasks like GSM8K and human evaluations in interactive settings
- Enhancing pre-training data quality and analyzing performance in code-switching scenarios to improve model's robustness and versatility
- Making models available to research community to develop more inclusive and efficient language processing technologies

## Summary

The report introduces \texttt{EEVE-Korean-v1.0}, a Korean adaptation of large language models that showcases impressive capabilities in understanding both English and Korean text. By leveraging the Efficient and Effective Vocabulary Expansion (EEVE) method, which involves parameter freezing and subword initialization, the EEVE-Korean-10.8B-v1.0 model has been able to excel in processing Korean language tasks while maintaining strong proficiency in English. This advancement has been achieved with just 2 billion tokens, highlighting a significant improvement in training efficiency and effectiveness for language models. Moving forward, the project aims to expand on this success by exploring the application of the vocabulary expansion methodology to additional languages to assess its generalizability and effectiveness. The goal is not only to broaden the linguistic range of the EEVE-Korean model but also to delve deeper into evaluating its reasoning and generative capabilities through diverse tasks such as complex mathematical reasoning tests like GSM8K and human evaluations in interactive settings like chatbots. Furthermore, future efforts will focus on enhancing pre-training data quality and analyzing performance in code-switching scenarios to refine the model's robustness and versatility. These initiatives are designed to broaden the model's applicability and efficacy, pushing the boundaries of what can be achieved with advanced language models. By making these models available to the research community, the project aims to contribute towards developing more inclusive and efficient language processing technologies that can benefit a wide range of users across different languages.