# Retrieval meets Long Context Large Language Models

**Authors:** Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, Bryan Catanzaro  
**arXiv:** [2310.03025](https://arxiv.org/abs/2310.03025)  
**Date:** 2023-10-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.03025v1/

## Key points

- Extending the context window of large language models (LLMs) is a popular trend among researchers to improve model performance.
- Retrieval-augmentation and long context extension are two methods compared in this study for enhancing LLM performance on downstream tasks.
- Advancements in GPU technology and memory-efficient attention mechanisms have made pretraining LLMs with longer context windows feasible.
- Fine-tuning existing models and introducing techniques like positional interpolation have led to significant improvements in model performance.
- Experimental results show that simple retrieval-augmentation can enhance the performance of 4K context LLMs, achieving comparable results to 16K long context models with less computation.
- Long context LLMs with extended window sizes can benefit from retrieval augmentation, suggesting a combination of both methods yields superior results on downstream tasks.

## Summary

Extending the Context Window of Large Language Models (LLMs): Enhancing Performance through Retrieval-Augmentation and Long Context Extension

In recent times, extending the context window of large language models (LLMs) has become a popular trend among researchers. This involves exploring different methods to improve the performance of these models. One such method is retrieval-augmentation, which has been around for years but is now being compared to the long context window approach. In this study, we aim to answer key questions regarding these two methods and their impact on downstream tasks. We delve into the evolution of long context LLMs and efficient attention methods in our related work section. With advancements in GPU technology and memory-efficient attention mechanisms, pretraining LLMs with longer context windows has become feasible. Researchers have pushed the boundaries by fine-tuning existing models or introducing new techniques like positional interpolation to extend context windows beyond traditional limits. These efforts have led to significant improvements in model performance on various tasks. Our experimental setup involves testing two state-of-the-art pretrained LLMs - a proprietary 43B GPT and LLaMA2-70B - on seven downstream long context tasks including question answering and query-based summarization. We find that simple retrieval-augmentation can significantly enhance the performance of 4K context LLMs, achieving comparable results to 16K long context models while requiring less computation. Additionally, our study demonstrates that long context LLMs with extended window sizes can still benefit from retrieval augmentation. Overall, our findings suggest that a combination of retrieval-augmentation and long context extension can yield superior results on downstream tasks compared to using either method alone. By providing insights into the effectiveness of these approaches, our study aims to guide practitioners in choosing the most suitable strategy for enhancing LLM performance.