# In-Context Retrieval-Augmented Language Models

**Authors:** Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham  
**arXiv:** [2302.00083](https://arxiv.org/abs/2302.00083)  
**Date:** 2023-01-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.00083v1/

## Key points

- Retrieval-Augmented Language Modeling (RALM) improves language modeling and provides a natural source attribution mechanism
- Existing RALM approaches require modifying the LM architecture, making deployment complicated
- In-Context RALM leaves the LM architecture unchanged and prepends grounding documents to the input
- In-context RALM using off-the-shelf general-purpose retrievers provides significant LM gains across model sizes and diverse corpora
- Document retrieval and ranking mechanisms can be specialized for the RALM setting to further boost performance
- In-context RALM employs a zero-effort document integration mechanism by simply prepending selected documents to the LM's input text
- Off-the-shelf retrievers lead to LM performance gains equivalent to increasing the LM's number of parameters by 2–3x across all examined text corpora
- Adapting document ranking to the LM task leads to further gains in the LM task corresponding to an additional size increase of 2x in the LM architecture
- In-context RALM has considerable potential for increasing the prevalence of LM grounding, particularly in settings where pretrained LMs must be used without modification or via API access

## Summary

Retrieval-Augmented Language Modeling (RALM) methods have been shown to improve language modeling and provide a natural source attribution mechanism by conditioning a language model on relevant documents from a grounding corpus during generation. However, existing RALM approaches require modifying the LM architecture, making deployment complicated. This paper proposes an alternative approach called In-Context RALM, which leaves the LM architecture unchanged and prepends grounding documents to the input. The authors demonstrate that in-context RALM using off-the-shelf general-purpose retrievers provides significant LM gains across model sizes and diverse corpora. They also show that document retrieval and ranking mechanisms can be specialized for the RALM setting to further boost performance. The paper presents experimental findings showing impressive performance gains with in-context RALM, even when working with off-the-shelf LMs through API access. The authors propose a simple but powerful framework for in-context RALM that employs a zero-effort document integration mechanism by simply prepending selected documents to the LM's input text. They evaluate their approach on five diverse corpora using open-source LMs ranging from 110M to 66B parameters. The authors evaluate the application of off-the-shelf retrievers to in-context RALM and find that it leads to LM performance gains equivalent to increasing the LM's number of parameters by 2–3x across all examined text corpora. They investigate methods for adapting document ranking to the LM task, leading to further gains in the LM task corresponding to an additional size increase of 2x in the LM architecture. In conclusion, this paper presents an under-explored alternative approach for incorporating external information into LMs through In-Context RALM without modifying their architectures significantly. The proposed method has considerable potential for increasing the prevalence of LM grounding, particularly in settings where pretrained LMs must be used without modification or via API access. The authors release all resources used for this paper to the community, hoping to drive further research on RALM and enable its wider adoption.