# Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

**Authors:** Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela  
**arXiv:** [2005.11401](https://arxiv.org/abs/2005.11401)  
**Date:** 2020-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2005.11401v4/

## Key points

- Large pre-trained language models have limitations in accessing and manipulating knowledge for knowledge-intensive NLP tasks
- Authors propose a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) models
- RAG models combine pre-trained parametric memory with non-parametric memory accessed through a pre-trained neural retriever
- Two formulations of RAG are compared: one with the same retrieved passages throughout the generated sequence, and another allowing different passages per token
- RAG models achieve state-of-the-art performance on three open domain question answering tasks, outperforming other models
- For language generation tasks, RAG models generate more specific, diverse, and factual language compared to baseline models
- Incorporating both parametric and non-parametric memory enhances the capabilities of pre-trained language models
- RAG models offer potential solutions for addressing limitations in accessing and manipulating knowledge effectively

## Summary

In this paper, the authors address the limitations of large pre-trained language models in accessing and manipulating knowledge for knowledge-intensive natural language processing (NLP) tasks. While these models have been successful in storing factual knowledge and achieving state-of-the-art results on downstream tasks, their ability to precisely manipulate knowledge is still limited compared to task-specific architectures. To overcome these limitations, the authors propose a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) models. These models combine pre-trained parametric memory (a seq2seq model) with non-parametric memory (a dense vector index of Wikipedia) accessed through a pre-trained neural retriever. The authors compare two formulations of RAG: one that conditions on the same retrieved passages throughout the generated sequence, and another that allows different passages per token. The authors evaluate their models on various knowledge-intensive NLP tasks and achieve state-of-the art performance on three open domain question answering tasks, outperforming both parametric seq2seq models and task specific retrieve and extract architectures. For language generation tasks, RAG models generate more specific, diverse, and factual language compared to a state of the art parametric only seq2seq baseline. Overall, this paper introduces a novel approach to enhancing the capabilities of pre trained language models by incorporating both parametric and non parametric memory. The proposed RAG models demonstrate improved performance on knowledge intensive NLP tasks and offer potential solutions for addressing the limitations of existing language models in accessing and manipulating knowledge effectively.