# Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

**Authors:** Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky  
**arXiv:** [2407.16833](https://arxiv.org/abs/2407.16833)  
**Date:** 2024-07-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.16833v1/

## Key points

- Compared performance of Retrieval Augmented Generation (RAG) and Long-Context (LC) Large Language Models (LLMs)
- LC consistently outperformed RAG in terms of average performance when adequately resourced
- RAG's lower cost remained a significant advantage
- Proposed Self-Route method to route queries to RAG or LC based on model self-reflection, maintaining comparable performance to LC while significantly reducing costs

## Summary

, , , , 

In our study, we compared the performance of Retrieval Augmented Generation (RAG) and Long-Context (LC) Large Language Models (LLMs), specifically Gemini-1.5 and GPT-4, on various public datasets for real and query-based tasks in English. We excluded summarization tasks without queries from our comparison. The datasets included NarrativeQA, Qasper, MultiFieldQA, HotpotQA, 2WikiMultihopQA, MuSiQue, QMSum from LongBench, and En.QA and EN.MC from ∞Bench. For evaluation metrics, we used F1 scores for open-ended QA tasks, accuracy for multi-choice QA tasks, and ROUGE score for summarization tasks. Our evaluation included three latest LLMs: Gemini-1.5-Pro supporting up to 1 million tokens, GPT-4O supporting 128k tokens, and GPT-3.5-Turbo supporting 16k tokens. Our results showed that LC consistently outperformed RAG in terms of average performance when adequately resourced. However,<kg>RAG's lower cost remained a significant advantage.</kg> To leverage the strengths of both approaches while reducing computation costs,<kg>we proposed Self-Route - a method that routes queries to RAG or LC based on model self-reflection.</kg> This approach maintained comparable performance to LC while significantly reducing costs.