# How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation

**Authors:** Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, Hany Hassan Awadalla  
**arXiv:** [2302.09210](https://arxiv.org/abs/2302.09210)  
**Date:** 2023-02-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.09210v1/

## Key points

- GPT models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated.
- A comprehensive evaluation of GPT models for machine translation was conducted in this study.
- The evaluation covered various aspects such as the quality of different GPT models in comparison with state-of-the-art research and commercial systems, the effect of prompting strategies, robustness towards domain shifts and document-level translation.
- Eighteen different translation directions involving high and low resource languages, as well as non-English-centric translations were experimented with, and the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002 were evaluated.
- The results showed that GPT models achieve very competitive translation quality for high resource languages while having limited capabilities for low resource languages.
- Hybrid approaches that combine GPT models with other translation systems can further enhance the translation quality.
- One limitation of this study is the inadequacy of current automatic evaluation metrics to capture the quality of GPT outputs accurately. Therefore, a comprehensive analysis was performed considering all metrics together along with human evaluation and qualitative analysis to cover a broad range of phenomena.
- It was recommended that readers consider the overall evaluations as a whole rather than relying solely on a specific metric to better understand the quality of GPT models' machine translation capabilities.
- These models may harbor language-specific biases and produce translations that perpetuate stereotypes and misinformation.
- Future work should focus on addressing these biases while also exploring ways to improve the performance of GPT models for low resource languages in machine translation tasks.
- Overall, this study provides valuable insights for researchers and practitioners in the field to better understand the potential and limitations of GPT models for machine translation.

## Summary

The latest Generative Pre-trained Transformer (GPT) models have shown remarkable capabilities for natural language generation, but their performance for machine translation has not been thoroughly investigated. To address this gap, a comprehensive evaluation of GPT models for machine translation was conducted in this study. The evaluation covered various aspects such as the quality of different GPT models in comparison with state-of-the-art research and commercial systems, the effect of prompting strategies, robustness towards domain shifts and document-level translation. Eighteen different translation directions involving high and low resource languages, as well as non-English-centric translations were experimented with, and the performance of three GPT models: ChatGPT, GPT3.5 (text-davinci-003), and text-davinci-002 were evaluated. The results showed that GPT models achieve very competitive translation quality for high resource languages while having limited capabilities for low resource languages. Hybrid approaches that combine GPT models with other translation systems can further enhance the translation quality. However, one limitation of this study is the inadequacy of current automatic evaluation metrics to capture the quality of GPT outputs accurately. Therefore, a comprehensive analysis was performed considering all metrics together along with human evaluation and qualitative analysis to cover a broad range of phenomena. It was recommended that readers consider the overall evaluations as a whole rather than relying solely on a specific metric to better understand the quality of GPT models' machine translation capabilities. Additionally, it was acknowledged that these models may harbor language-specific biases and produce translations that perpetuate stereotypes and misinformation. Future work should focus on addressing these biases while also exploring ways to improve the performance of GPT models for low resource languages in machine translation tasks. Overall, this study provides valuable insights for researchers and practitioners in the field to better understand the potential and limitations of GPT models for machine translation.