# GAIA: a benchmark for General AI Assistants

**Authors:** Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom  
**arXiv:** [2311.12983](https://arxiv.org/abs/2311.12983)  
**Date:** 2023-11-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.12983v1/

## Key points

- GAIA is a benchmark for General AI Assistants in AI research
- GAIA presents real-world questions that require reasoning, multi-modality handling, web browsing, and tool-use proficiency
- Human respondents achieve a 92% accuracy rate compared to only 15% for GPT-4 equipped with plugins
- GAIA's philosophy diverges from the prevailing approach in AI benchmarks by targeting tasks that are conceptually simple yet challenging for advanced AIs
- The development of Artificial General Intelligence (AGI) depends on a system's ability to exhibit similar robustness as an average human when answering such questions
- 466 questions and their corresponding answers have been devised using GAIA's methodology
- The authors release all 466 questions but retain the answers to 300 of them for a leaderboard accessible at https://huggingface.co/gaia-benchmark
- GAIA provides an important benchmark for evaluating General AI Assistants' capabilities by challenging them with real-world questions that require fundamental cognitive abilities

## Summary

The paper introduces GAIA, a benchmark for General AI Assistants that aims to represent a significant milestone in AI research. GAIA presents real-world questions that require fundamental abilities such as reasoning, multi-modality handling, web browsing, and tool-use proficiency. While these questions are conceptually simple for humans, they pose a challenge for most advanced AIs. The authors demonstrate that human respondents achieve a 92% accuracy rate compared to only 15% for GPT-4 equipped with plugins. This performance disparity is noteworthy considering the recent trend of Language and Learning Models (LLMs) outperforming humans on tasks requiring professional skills in fields like law or chemistry. GAIA's philosophy diverges from the prevailing approach in AI benchmarks which focuses on targeting tasks that are increasingly difficult for humans. The authors argue that the development of Artificial General Intelligence (AGI) depends on a system's ability to exhibit similar robustness as an average human when answering such questions. To facilitate this research, the authors have devised 466 questions and their corresponding answers using GAIA's methodology. While the authors release all 466 questions, they retain the answers to 300 of them to power a leaderboard accessible at https://huggingface.co/gaia-benchmark. This leaderboard allows researchers to compare their AI systems' performance against others in tackling the GAIA benchmark. In conclusion, GAIA provides an important benchmark for evaluating General AI Assistants' capabilities by challenging them with real-world questions that require fundamental cognitive abilities. By focusing on tasks that are conceptually simple yet challenging for advanced AIs, GAIA aims to drive progress towards achieving Artificial General Intelligence.