# Survey on Evaluation of LLM-based Agents

**Authors:** Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, Michal Shmueli-Scheuer  
**arXiv:** [2503.16416](https://arxiv.org/abs/2503.16416)  
**Date:** 2025-03-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.16416v1/

## Key points

- LLM-based agents have revolutionized AI by enabling autonomous systems to plan, reason, use tools, and maintain memory in dynamic environments.
- Evaluation methodologies for advanced agents focus on four critical dimensions: fundamental agent capabilities, application-specific benchmarks, benchmarks for generalist agents, and frameworks for evaluating agents.
- Specialized benchmarks like those introduced by Huet et al. (2025) and Wu et al. (2024a) assess episodic memory generation and management in LLMs.
- Memory mechanisms improve real-time decision-making and learning in agent settings, as demonstrated by Reflexion (Shinn et al., 2023), RAISE (Liu et al., 2024a), and KARMA (Wang et al., 2024b).
- Agent benchmarks provide a systematic framework for assessing the diverse capabilities of LLM-based agents across various applications.
- The analysis highlights trends towards more realistic evaluations with continuously updated benchmarks but also identifies gaps in assessing cost-efficiency, safety, robustness, and scalability of evaluation methods.

## Summary

, , , , 

The emergence of LLM-based agents has revolutionized the field of AI, enabling autonomous systems to plan, reason, use tools, and maintain memory while interacting with dynamic environments. This paper presents a comprehensive survey of evaluation methodologies for these advanced agents, focusing on four critical dimensions: fundamental agent capabilities, application-specific benchmarks for web, software engineering, scientific, and conversational agents, benchmarks for generalist agents, and frameworks for evaluating agents. For episodic memory evaluation, (Huet et al., 2025) introduces a specialized benchmark to assess how LLMs generate and manage memories capturing specific events with contextual details. StreamBench (Wu et al., 2024a) evaluates how agents leverage external memory components to continuously improve performance over time across diverse datasets. Memory mechanisms also enhance real-time decision-making and learning in agent settings. Reflexion (Shinn et al., 2023), RAISE (Liu et al., 2024a), and KARMA (Wang et al., 2024b) demonstrate how memory systems significantly improve agent performance in tasks requiring complex reasoning and information retention. The landscape of application-specific agents is expanding rapidly across categories such as tools, web, software, game, embodied, and scientific agents. Agent benchmarks offer a systematic framework for assessing the diverse capabilities of LLM-based agents by integrating clearly defined tasks with evaluation strategies tailored to their unique applications. Furthermore, the paper discusses benchmarks and leaderboards for evaluating general-purpose agents that assess the agent's ability to perform tasks efficiently with operational viability. The analysis reveals emerging trends towards more realistic evaluations with continuously updated benchmarks but also highlights critical gaps in assessing cost-efficiency, safety, robustness, and developing scalable evaluation methods. In conclusion, this survey maps the evolving landscape of agent evaluation methodologies while identifying current limitations and proposing directions for future research to address these challenges effectively.