# How do Large Language Models Handle Multilingualism?

**Authors:** Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, Lidong Bing  
**arXiv:** [2402.18815](https://arxiv.org/abs/2402.18815)  
**Date:** 2024-02-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18815v1/

## Key points

- Recent advancements in large language models (LLMs) such as PaLM, GPT-4, LLaMA, and Mistral have revolutionized natural language processing tasks
- These models are extensively pre-trained on massive corpora containing various languages and exhibit exceptional capabilities in understanding and generating text across multiple languages
- The intricate workings of their multilingual processing mechanisms remain largely unclear, leading to the research question: How do large language models handle multilingualism?
- Prior studies have explored cross-lingual performance and structural commonalities between languages, while recent investigations focus on specific model architectures like the Transformer architecture to understand reasoning abilities with self-attention layers or the role of feed-forward layers in storing factual knowledge
- Analyzing decoded embeddings after each layer reveals that LLMs convert multilingual inputs into English at initial layers for task-solving before incorporating multilingual knowledge for problem-solving in intermediate layers, with final layers generating responses aligned with the original language of the query
- A novel Parallel Language specific Neuron Detection (PLND) method is introduced to measure neuron significance without labels, validating efficient handling of multilingual inputs through comprehensive ablation analysis by deactivating neurons across different layers and structures
- There is a need for a more holistic understanding of LLMs' multilingual mechanisms to enhance their capabilities with minimal training effort, emphasizing the importance of exploring decoded embeddings and token distributions among layers when processing non-English instructions

## Summary

Recent advancements in large language models (LLMs) such as PaLM, GPT-4, LLaMA, and Mistral have revolutionized natural language processing tasks by seamlessly integrating into daily and professional uses. These models are extensively pre-trained on massive corpora containing various languages and exhibit exceptional capabilities in understanding and generating text across multiple languages. Despite their proven effectiveness, the intricate workings of their multilingual processing mechanisms remain largely unclear. This prompts the research question: How do large language models handle multilingualism? Prior studies have explored the multilingual capabilities of language models, focusing on cross-lingual performance or structural commonalities between languages. However, more recent investigations delve into specific model architectures like the Transformer architecture to understand reasoning abilities with self-attention layers or the role of feed-forward layers in storing factual knowledge. In this study, we aim to gain a deeper understanding of how LLMs handle multilingual inputs by analyzing decoded embeddings after each layer when processing non-English instructions. By classifying these embeddings as English or non-English tokens, we observe how LLMs convert multilingual inputs into English at initial layers for task-solving before incorporating multilingual knowledge for problem-solving in intermediate layers. The final layers then generate responses aligned with the original language of the query. To further investigate the presence of language-specific neurons during processing different languages, we introduce a novel Parallel Language specific Neuron Detection (PLND) method that effectively measures neuron significance without labels. Through comprehensive ablation analysis by deactivating neurons across different layers and structures, we validate our proposed framework for handling multilingual inputs efficiently. Moreover, recent research developments highlight the need for a more holistic understanding of LLMs' multilingual mechanisms to enhance their capabilities with minimal training effort. By exploring decoded embeddings and token distributions among layers when processing non-English instructions, we present a detailed insight into the intricate workflow of LLMs when handling diverse linguistic inputs. This study contributes to bridging gaps in existing literature and offers valuable insights into optimizing large language models for enhanced multilingual performance.