# BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text

**Authors:** Hadeel Saadany, Constantin Orasan  
**arXiv:** [2109.14250](https://arxiv.org/abs/2109.14250)  
**Date:** 2021-09-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.14250v1/

## Key points

- The research explores the effectiveness of automatic quality metrics in detecting sentiment-critical machine translation errors.
- Three canonical metrics (BLEU, METEOR, and BERTScore) are compared in terms of their performance on meaningless translations and meaningful translations with critical errors that distort sentiment.
- Current metrics are not sensitive enough to penalize sentiment-critical errors and can even give high scores to mistranslations.
- Fine-tuning these metrics is necessary to accurately assess sentiment-oriented text.
- Developing a sentiment-targeted evaluation measure is suggested to address this issue.
- BLEU compares n-grams between candidate and reference translations, while METEOR incorporates semantic information and applies importance weighting.
- Both BLEU and METEOR have limitations in assessing sentiment-critical errors.
- Improving quality metrics to capture sentiment-critical lexicon is emphasized for enhancing their performance with sentiment-oriented text.

## Summary

In this research, the authors explore the effectiveness of automatic quality metrics in detecting sentiment-critical machine translation errors that distort user-generated content (UGC). They compare the performance of three canonical metrics – BLEU, METEOR and BERTScore – on both meaningless translations and meaningful translations with a critical error that exclusively distorts the sentiment. The results show that current metrics are not sensitive enough to penalize sentiment-critical errors and can even give high scores to mistranslations. This highlights the need for fine-tuning these metrics to accurately assess sentiment-oriented text. The authors suggest developing a sentiment-targeted evaluation measure to address this issue. BLEU compares n-grams between candidate and reference translations, while METEOR incorporates semantic information and applies importance weighting. However, both metrics have limitations in assessing sentiment-critical errors. The authors emphasize the importance of improving quality metrics to capture sentiment-critical lexicon and enhance their performance with sentiment-oriented text.