# What do Large Language Models Need for Machine Translation Evaluation?

**Authors:** Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orăsan, Tharindu Ranasinghe, Frédéric Blain  
**arXiv:** [2410.03278](https://arxiv.org/abs/2410.03278)  
**Date:** 2024-10-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.03278v1/

## Key points

- Large language models (LLMs) have improved applications in natural language processing tasks like machine translation, text summarization, and information retrieval.
- Enhanced natural language understanding capabilities and contextual awareness contribute to the advancements of LLMs.
- Traditional approaches for evaluating machine translation quality include metrics like BLEU and fine-tuning multilingual pre-trained language models (PTLMs) on human evaluation data.
- Recent studies have explored prompting techniques to instruct LLMs for translation quality evaluation without training.
- Reference translations are crucial for accurate evaluations using LLMs across different languages.
- Larger LLM variants tend to benefit more from Chain-of-Thought (CoT) prompting than smaller ones, but model size doesn't always correlate with performance.
- A 7-billion parameter model outperformed other models for most languages, indicating that larger size doesn't always mean better performance.
- COMET models fine-tuned on multilingual data generally outperformed LLM prompting results in MT quality evaluation.
- The study emphasizes the importance of reference translations and provides insights into factors influencing LLM-based MT evaluation.

## Summary

The use of large language models (LLMs) has greatly improved their applications in natural language processing tasks, such as machine translation (MT), text summarization, and information retrieval. These advancements are attributed to the enhanced natural language understanding capabilities and contextual awareness of LLMs. Traditional approaches for evaluating MT quality rely on metrics like BLEU or fine-tuning multilingual pre-trained language models (PTLMs) on human evaluation data. However, recent studies have explored prompting techniques to instruct LLMs for translation quality evaluation. This study investigates the ability of LLMs in evaluating MT quality without training across eight different languages and suggests that reference translations play a crucial role in accurate evaluations. Additionally, larger LLM variants tend to benefit more from Chain-of-Thought (CoT) prompting than smaller ones. Surprisingly, a 7-billion parameter model outperformed other models for most languages, indicating that model size does not always correlate with performance. Furthermore, COMET models fine-tuned on multilingual data generally outperformed LLM prompting results. Overall, this study highlights the importance of reference translations and provides valuable insights into factors influencing LLM-based MT evaluation. The release of prompt templates, code, and data aims to facilitate reproducibility and further research in this area. The recent surge in the use of large language models (LLMs) has significantly improved their applications in various downstream tasks such as natural language processing (NLP), including machine translation (MT), text summarization, and information retrieval. These advancements are attributed to the enhanced natural language understanding capabilities and contextual awareness of LLMs, as well as their versatile knowledge base. Traditional approaches for evaluating MT quality rely on metrics like BLEU, BLEURT, or BERTScore to compare the MT output with a reference translation. In the absence of reference translations, quality estimation methods such as fine-tuning multilingual pre-trained language models (PTLMs) on human evaluation data are often used. However, recent studies have explored prompting techniques to instruct LLMs in outputting scores for translation quality evaluation. This study investigates the ability of LLMs to evaluate MT quality without training across eight different languages covering high-, medium-, and low-resource languages. The findings suggest that reference translations play a crucial role in achieving accurate evaluations using LLMs. While larger models do not always outperform smaller ones, they tend to benefit more from Chain-of-Thought (CoT) prompting than smaller model variants. Surprisingly, a 7-billion parameter model surpassed other models for most language pairs, indicating that model size does not always correlate with performance. Additionally, COMET models fine-tuned on multilingual data generally outperformed LLM prompting results. Overall, this study provides valuable insights into the factors influencing LLM-based MT evaluation and highlights the importance of reference translations in achieving accurate evaluations. The release of prompt templates, code, and data aims to facilitate reproducibility and further research in this area. Future directions may involve exploring additional prompting techniques and investigating the performance of larger LLM variants with higher computational costs.