# RA-DIT: Retrieval-Augmented Dual Instruction Tuning

**Authors:** Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, Scott Yih  
**arXiv:** [2310.01352](https://arxiv.org/abs/2310.01352)  
**Date:** 2023-10-02  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.01352v3/

## Key points

- Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores.
- Existing approaches for building RALMs either require expensive modifications to LM pre-training or use post-hoc integration of the data store, leading to suboptimal performance.
- The authors introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that retrofit any LLM with retrieval capabilities.
- RA-DIT operates in two distinct fine-tuning steps: updating the pre-trained LM to better utilize retrieved information and updating the retriever to return more relevant results preferred by the LM.
- Fine-tuning over tasks that require both knowledge utilization and contextual awareness leads to significant performance improvements at each stage, with additional gains when using both stages.
- The best model proposed in this study, RA-DIT 65B, achieves state-of-the-art performance across various knowledge intensive zero and few shot learning benchmarks.
- RA-DIT 65B outperforms existing in context RALM approaches by up to +8.9% in the 0 shot setting and +1.4% in the 5 shot setting on average.
- The paper provides additional details on their approach and experimental results for commonsense reasoning tasks.
- The authors also discuss related work in this area.

## Summary

Retrieval-augmented language models (RALMs) have shown promise in improving performance by accessing long-tail and up-to-date knowledge from external data stores. However, building RALMs is challenging, as existing approaches either require expensive modifications to LM pre-training or use post-hoc integration of the data store, leading to suboptimal performance. In this paper, the authors introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that offers a third option by retrofitting any LLM with retrieval capabilities. RA-DIT operates in two distinct fine-tuning steps: the first step updates a pre-trained LM to better utilize retrieved information, while the second step updates the retriever to return more relevant results preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, the authors demonstrate significant performance improvements at each stage. Furthermore, using both stages leads to additional gains. The best model proposed in this study, RA-DIT 65B, achieves state-of-the-art performance across various knowledge intensive zero and few shot learning benchmarks. It outperforms existing in context RALM approaches by up to +8.9% in the 0 shot setting and +1.4% in the 5 shot setting on average. The paper provides additional details on their approach and experimental results for commonsense reasoning tasks. The authors also discuss related work in this area.