# PersonaGym: Evaluating Persona Agents and LLMs

**Authors:** Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, Vishvak Murahari  
**arXiv:** [2407.18416](https://arxiv.org/abs/2407.18416)  
**Date:** 2024-07-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.18416v2/

## Key points

- PersonaGym: a dynamic evaluation framework for assessing persona agents in large language models (LLMs)
- Normative Evaluation
- Prescriptive Evaluation
- Descriptive Evaluation
- Tasks focus on aspects such as optimal decision-making, linguistic habits adherence, persona consistency, toxicity control, and action justification
- Introduction of PersonaScore: an automated metric to quantify persona agent capabilities across the evaluation tasks
- Benchmarking of PersonaScore on 200 persona agents from six LLMs reveals room for improvement in abilities
- Complexity of model does not guarantee enhanced persona agent abilities; example with GPT 3.5 Sonnet and Claude 3.5 Sonnet
- Need for algorithmic and architectural advancements to develop more faithful and performant persona agents across various applications like education, healthcare, and entertainment

## Summary

In this study, the researchers introduce PersonaGym - a groundbreaking dynamic evaluation framework designed to assess persona agents within large language models (LLMs). The framework consists of five key evaluation tasks grounded in decision theory. These tasks aim to understand how agents interact with their environment and make decisions based on their goals and beliefs. The five evaluation tasks include Normative Evaluation, Prescriptive Evaluation, and Descriptive Evaluation. Each task focuses on different aspects of agent behavior such as optimal decision-making, adherence to linguistic habits, consistency of persona, control of toxicity, and justification of actions. To quantify the capabilities of persona agents across these five evaluation tasks, the researchers also introduce PersonaScore - an automated metric. By benchmarking the PersonaScore of 200 persona agents from six open and closed-source LLMs on 10,000 agent-relevant questions, the study reveals significant opportunities for improvement in persona agent abilities. Surprisingly, the findings show that model complexity does not necessarily guarantee enhanced persona agent abilities. For example, despite being a more advanced model than GPT 3.5 Sonnet, Claude 3.5 Sonnet only exhibits a 2.97% relative improvement in PersonaScore. Overall, this research highlights the pressing need for algorithmic and architectural advancements to develop more faithful and performant persona agents across various applications such as education, healthcare, and entertainment. By providing a comprehensive evaluation framework like PersonaGym and an automated metric like PersonaScore grounded in decision theory principles,this study sets a new standard for assessing persona agent performance in LLMs and paves the way for future advancements in this field.