# Seven Failure Points When Engineering a Retrieval Augmented Generation System

**Authors:** Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, Mohamed Abdelrazek  
**arXiv:** [2401.05856](https://arxiv.org/abs/2401.05856)  
**Date:** 2024-01-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.05856v1/

## Key points

- Growing trend towards incorporating semantic search capabilities into applications through RAG systems
- RAG systems match query documents with semantic relevance using large language models like ChatGPT
- Primary goals of RAG systems: mitigate hallucinated responses, connect sources to answers, eliminate need for document annotation
- Challenges faced by RAG systems: information retrieval limitations, reliance on LLMs
- Insightful experience report based on three case studies in research, education, and biomedical fields
- Seven critical failure points in designing RAG systems identified through case studies
- Validation of a RAG system feasible during operation, robustness evolves over time
- Potential research directions outlined to enhance RAG system performance

## Summary

In the realm of software engineering, there is a growing trend towards incorporating semantic search capabilities into applications through the implementation of Retrieval Augmented Generation (RAG) systems. These systems involve matching query documents with semantic relevance and utilizing large language models (LLMs) like ChatGPT to extract accurate responses. The primary goals of RAG systems are to mitigate issues such as hallucinated responses from LLMs, connect sources to generated answers, and eliminate the need for document annotation. However, despite their potential benefits, RAG systems face inherent limitations associated with information retrieval and reliance on LLMs. To shed light on these challenges, this paper presents an insightful experience report based on three case studies spanning diverse domains including research, education, and biomedical fields. Through these case studies, seven critical failure points in designing RAG systems have been identified. The key takeaways from this work emphasize that validating a RAG system is only feasible during operation and that the system's robustness evolves over time rather than being predetermined at the outset. Furthermore, the paper outlines potential research directions for the software engineering community to enhance RAG system performance. Delving deeper into specific case studies discussed in the paper
1. Cognitive Reviewer: A RAG system tailored for researchers to analyze scientific documents by ranking them according to specified objectives. This system aids PhD students at Deakin University in conducting literature reviews by enabling direct questioning against uploaded documents. 2. AI Tutor: Another RAG system designed to assist students in querying unit-related information sourced from learning content. Through these case studies and analysis of failure points such as missing content, document ranking discrepancies, context consolidation limitations, extraction errors, and formatting issues within RAG systems, valuable insights are gleaned for optimizing future implementations. By addressing these challenges and leveraging lessons learned from real-world scenarios across various domains, software engineers can refine their approach towards designing more effective and reliable RAG systems.