# Long Context vs. RAG for LLMs: An Evaluation and Revisits

**Authors:** Xinze Li, Yixin Cao, Yubo Ma, Aixin Sun  
**arXiv:** [2501.01880](https://arxiv.org/abs/2501.01880)  
**Date:** 2024-12-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.01880v1/

## Key points

- Growing interest in enhancing capabilities of Large Language Models (LLMs) with long external contexts
- Two main strategies: Long Context (LC) and Retrieval-Augmented Generation (RAG)
- Notable advancement in retrieval methods: RAPTOR improves accuracy by generating recursive summaries in a tree structure
- Various LLM models excel in specialized areas such as reasoning efficiency, conversational understanding, text summarization, knowledge understanding, multilingual translation, mathematical computations, and logical reasoning
- Trend towards increasing context length in newly released models categorized as short (up to 4K), long (up to 32K), and ultra-long (more than 32K) context models
- Advancements offer potential for handling complex questions requiring information synthesis from multiple document parts
- Importance of considering context relevance when optimizing LLMs with external knowledge sources
- Need for tailored approaches based on specific task requirements and further research for more effective utilization of external knowledge sources

## Summary

In recent years, there has been a growing interest in enhancing the capabilities of Large Language Models (LLMs) to incorporate extremely long external contexts. Two main strategies have emerged: Extending context windows, known as Long Context (LC), and using retrievers to selectively access relevant information, known as Retrieval-Augmented Generation (RAG). This paper delves into recent studies on this topic, shedding light on key insights and discrepancies. One notable advancement in retrieval methods is RAPTOR (Sarthi et al., 2024), which improves accuracy by generating recursive summaries of text chunks organized in a tree structure. By summarizing text segments at various levels and forming a hierarchical tree representing the document's content, RAPTOR enables retrieval models to extract context at varying levels of detail. This method enhances retrieval accuracy for tasks requiring long-range or multi-step reasoning. When it comes to LLMs with extended context capabilities, various models excel in specialized areas. For instance, ChatGLM2-6B-32K focuses on high reasoning efficiency with low memory usage, while XGen-7B-8K enhances conversational understanding and text summarization. InternLM-7B-8k is optimized for knowledge understanding and multilingual translation, while other models like DeepSeek-V2-Chat, Qwen2-72B-Instruct, Mixtral-7x8b, and DBRX-Instruct excel in mathematical computations and logical reasoning. There is a clear trend towards increasing context length in newly released models. These models are categorized based on their supported context windows: short (up to 4K), long (up to 32K), and ultra-long (more than 32K) context models. The advancements in LLMs with extended context capabilities offer significant potential for handling complex questions that require synthesizing information from multiple parts of a document. In conclusion, the trade-offs between RAG and LC strategies underscore the importance of considering context relevance when optimizing LLMs with external knowledge sources. The diverse capabilities of different LLM models highlight the need for tailored approaches based on specific task requirements. Further research in this area can lead to more effective utilization of external knowledge sources for enhancing LLM performance across various applications.