# ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

**Authors:** Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, Thien Huu Nguyen  
**arXiv:** [2304.05613](https://arxiv.org/abs/2304.05613)  
**Date:** 2023-04-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.05613v1/

## Key points

- Large language models (LLMs) are a significant breakthrough in natural language processing (NLP)
- ChatGPT is an impressive LLM system for language generation that has attracted public attention
- It remains unclear whether ChatGPT can be applied effectively to other languages or if more language-specific technologies are necessary
- Researchers conducted experiments covering 37 different languages with high, medium, low, and extremely low resources to evaluate ChatGPT's performance on multiple tasks with diverse languages and large datasets
- Results showed that ChatGPT's performance was consistently inferior to mT5-XXL's performance for summarization tasks across different languages due to its tendency to generate lengthy summaries
- Success rates for lower-resource languages were lower than those for higher-resource languages, indicating that more language-specific technologies may be needed when applying ChatGPT to other languages
- The study calls for further research to develop better models and understanding of multilingual learning in order to improve the effectiveness of LLMs such as ChatGPT across various languages and tasks.

## Summary

Large language models (LLMs) have become a significant breakthrough in natural language processing (NLP) in recent years. Among the most exciting LLM systems developed is ChatGPT, which has impressive skills for language generation and has attracted public attention. While ChatGPT has been successful in English, it remains unclear whether it can be applied effectively to other languages or if more language-specific technologies are necessary. To evaluate ChatGPT's performance on multiple tasks with diverse languages and large datasets, researchers conducted experiments covering 37 different languages with high, medium, low, and extremely low resources. The study focused on zero-shot learning settings to improve reproducibility and simulate interactions of general users. Results showed that ChatGPT's performance was consistently inferior to mT5-XXL's performance for summarization tasks across different languages due to its tendency to generate lengthy summaries. Additionally, success rates for lower-resource languages were lower than those for higher-resource languages. This indicates that more language-specific technologies may be needed when applying ChatGPT to other languages. The study calls for further research to develop better models and understanding of multilingual learning in order to improve the effectiveness of LLMs such as ChatGPT across various languages and tasks.