# RAGAS: Automated Evaluation of Retrieval Augmented Generation

**Authors:** Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert  
**arXiv:** [2309.15217](https://arxiv.org/abs/2309.15217)  
**Date:** 2023-09-26  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2309.15217v1/

## Key points

- RAGAs is a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines
- RAG systems consist of retrieval and LLM based generation modules
- Evaluating RAG architectures is challenging due to multiple dimensions that need to be considered
- RAGAs propose metrics to evaluate the ability of retrieval systems, faithful exploitation of passages by LLMs, and quality of generated output
- RAGAs eliminate the need for human annotations, enabling faster evaluation cycles for RAG architectures
- The proposed framework addresses challenges in evaluating RAG systems and contributes to improving their performance
- Efficient evaluation methods like RAGAs are crucial with the increasing adoption of LLMs in various applications.

## Summary

We introduce RAGAs (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation (RAG) pipelines. RAG systems consist of a retrieval and an LLM based generation module, which provide LLMs with knowledge from a reference textual database. This enables them to act as a natural language layer between a user and textual databases, reducing the risk of hallucinations. However, evaluating RAG architectures is challenging due to multiple dimensions that need to be considered, such as the ability of the retrieval system to identify relevant context passages, the ability of the LLM to use these passages faithfully, and the quality of the generated output. To address this challenge, we propose RAGAs as a suite of metrics that can evaluate these different dimensions without relying on ground truth human annotations. By providing a framework for faster evaluation cycles of RAG architectures, especially given the fast adoption of LLMs, RAGAs can significantly contribute to improving these systems. In recent years, Language Models (LMs) have become repositories of vast knowledge about the world. They are capable of answering questions without accessing external sources. However, LMs have limitations in answering questions about events that occurred after their training and struggle with memorizing rarely mentioned knowledge in their training corpus. To overcome these limitations, Retrieval Augmented Generation (RAG) approaches have been introduced. These approaches involve retrieving relevant passages from a corpus and feeding them along with the question to an LM for generating answers. While retrieval-augmented strategies have proven useful, their implementation requires tuning and performance evaluation across various dimensions. The existing methods rely on ground truth human annotations for evaluation, which can be time-consuming and impractical in large-scale applications. In this paper, we propose RAGAs as a reference-free evaluation framework for RAG pipelines. It provides metrics that assess the ability of retrieval systems to identify relevant context passages, the faithful exploitation of these passages by the LLM, and the quality of the generated output. Importantly, RAGAs eliminate the need for human annotations, enabling faster evaluation cycles for RAG architectures. The proposed framework addresses the challenges in evaluating RAG systems and contributes to improving their performance. With the increasing adoption of LLMs in various applications, such as question answering and information retrieval, efficient evaluation methods like RAGAs are crucial. By providing a suite of metrics that can assess different dimensions of RAG architectures without relying on human annotations, RAGAs enable researchers and practitioners to evaluate and refine these systems more efficiently.