# LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

**Authors:** Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, Yiqun Liu  
**arXiv:** [2412.05579](https://arxiv.org/abs/2412.05579)  
**Date:** 2024-12-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.05579v2/

## Key points

- The paper explores the concept of "LLMs-as-judges" in natural language responses
- LLM judges are effective, have task generalization capabilities, and offer interpretability through natural language
- The survey covers five key perspectives: Functionality, Methodology, Applications, Meta-evaluation, and Limitations
- It defines LLMs-as-Judges and explains their advantages
- Discusses the methodology for constructing an evaluation system with LLMs
- Explores potential domains where LLM judges can be applied and methods for evaluating them
- Analyzes limitations associated with LLM judges and suggests future directions for the field

## Summary

The paper titled "LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods" by authors Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu delves into the rapidly evolving landscape of Large Language Models (LLMs) and their application as evaluators in natural language responses. This innovative framework known as ''LLMs-as-judges'' has garnered significant attention from academia and industry for its remarkable effectiveness, task generalization capabilities, and interpretability through natural language. The comprehensive survey presented in the paper explores the LLMs-as-judges paradigm from five key perspectives: Functionality, Methodology, Applications, Meta-evaluation, and Limitations. The authors systematically define LLMs-as-Judges and delve into their functionality to elucidate why utilizing LLM judges is advantageous. They also address the methodology required to construct an evaluation system with LLMs and discuss how these judges can be effectively employed. Furthermore, the paper investigates potential domains where LLM judges can be applied and examines various methods for evaluating them across different contexts. The authors provide a detailed analysis of the limitations associated with LLM judges while also discussing potential future directions for this burgeoning field. Through a structured and insightful analysis, this paper aims to offer valuable insights into the development and application of LLMs-as-judges in both research settings and practical applications. The authors have committed to maintaining an updated resource list related to this topic at https://github.com/CSHaitao/Awesome-LLMs-as-Judges. With 60 pages of comprehensive content that is continuously updated, this paper serves as a valuable resource for researchers and practitioners interested in leveraging LLM-based evaluation methods for various applications in natural language processing.