# Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator

**Authors:** Frederic Kirstein, Terry Ruas, Bela Gipp  
**arXiv:** [2411.18444](https://arxiv.org/abs/2411.18444)  
**Date:** 2024-11-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.18444v1/

## Key points

- Meeting summaries are crucial in professional environments for providing references, updates for absentees, and reinforcing key topics discussed during meetings.
- The integration of summarization services into popular digital meeting platforms has increased the importance of generating high-quality meeting summaries.
- Traditional evaluation of meeting summaries is done through costly and time-consuming human assessment.
- Existing automatic metrics like ROUGE, BERTScore, and BARTScore show low correlation with human judgment in evaluating meeting summaries.
- MESA (Meeting Summary Assessor) is a multi-stage LLM-based framework designed to mimic human evaluation approaches for meeting summary assessment.

## Summary

, , , , 

In this study, we address the challenge of evaluating meeting summaries automatically using natural language generation (NLG) systems. Meeting summaries play a crucial role in professional environments by providing references, updates for absentees, and reinforcing key topics discussed during meetings. With the integration of summarization services into popular digital meeting platforms like Zoom, Microsoft Teams, and Google Meet, the importance of generating high-quality meeting summaries has increased significantly. The evaluation of generated meeting summaries is traditionally done through costly and time-consuming human assessment. To overcome this limitation, there is a need for an automatic evaluator that can provide insights along with scoring to enable sophisticated techniques such as feedback-based summary refinement and reinforcement learning from AI feedback. However, established automatic metrics like ROUGE, BERTScore, and BARTScore show a relatively low correlation with human judgment and often fail to detect errors accurately or reflect error impact on summary quality. Large language models (LLMs) have emerged as potential evaluators for text summarization tasks by assigning Likert scores based on predefined guidelines. However, existing LLM-based approaches face limitations in meeting summarization contexts due to oversight of typical errors such as structural presentation issues and coreference problems. Moreover, subjective annotation guidelines may lead to inconsistent interpretations by LLMs, resulting in unreliable evaluations. To address these challenges, we introduce MESA (Meeting Summary Assessor), a multi-stage LLM-based framework designed to mimic human evaluation approaches. MESA operates on three levels: error detection at the individual error type level, multi-agent discussion for decision refinement, and feedback-based self-training to improve error definition understanding and alignment with human judgment. By leveraging GPT-4o as its backbone model, MESA achieves mid to high correlations with human judgment in error detection and reflects error impact on summary quality more accurately than previous methods. Overall, MESA's flexibility in adapting to custom error guidelines makes it suitable for various tasks with limited human-labeled data. The framework enables thorough error detection, consistent rating, and adaptability to custom error definitions while providing valuable insights into meeting summary quality assessment without relying solely on costly human evaluation methods.