# TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

**Authors:** Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou  
**arXiv:** [2412.14161](https://arxiv.org/abs/2412.14161)  
**Date:** 2024-12-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.14161v1/

## Key points

- Interactions with computers are integral in personal and professional lives in the digital age
- Large language models (LLMs) have enabled rapid evolution of AI agents to perform work-related tasks
- TheAgentCompany benchmark assesses performance of LLM agents in a simulated professional environment
- Competitive agent autonomously completed 24% of tasks, indicating effectiveness for simpler tasks but challenges for complex ones
- TheAgentCompany offers a comprehensive evaluation framework for AI agents interacting like human workers
- Collaborative effort behind TheAgentCompany involved multiple institutions and individuals contributing to its development
- Valuable tool for assessing AI agent performance in real-world work scenarios and advancing research in AI automation

## Summary

, , , , 

In today's digital age, our interactions with computers have become an integral part of both our personal and professional lives. With the advancements in large language models (LLMs), artificial intelligence (AI) agents have rapidly evolved to interact with and influence their environments. The question arises: how effective are these AI agents in accelerating or autonomously performing work-related tasks? This inquiry holds significant implications for industries considering AI integration into their workflows and for policymakers seeking to understand the impact on the labor market. To assess the performance of LLM agents in real-world professional tasks, a new benchmark called TheAgentCompany has been introduced. This benchmark simulates a digital worker's activities within a small software company environment, including web browsing, coding, program execution, and communication with colleagues. Baseline agents powered by closed API-based and open-weights language models were tested within this environment. The results revealed that the most competitive agent was able to autonomously complete 24% of the tasks. This nuanced finding suggests that while simpler tasks can be automated effectively by current systems, more complex long-horizon tasks still pose challenges. TheAgentCompany offers a comprehensive evaluation framework for AI agents interacting with the world like human workers. Furthermore, comparisons were drawn between TheAgentCompany and other existing benchmarks in terms of task diversity, realism, interface capabilities, self-hosted environments, interaction requirements, checkpoint evaluations, and NPC agent interactions. The collaborative effort behind TheAgentCompany involved multiple institutions and individuals contributing to task design, infrastructure development, experiments, Sotopia integration, task development, ideation discussions formulation under the guidance of project leads. Acknowledgments were extended to Open Philanthropy for funding support and various individuals for insightful discussions throughout the project. Overall, TheAgentCompany presents a valuable tool for assessing AI agent performance in real-world work scenarios and contributes to advancing research in AI automation within professional settings.