# RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards

**Authors:** Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu, Zhiyuan Liu, Maosong Sun, Chenyan Xiong  
**arXiv:** [2410.13509](https://arxiv.org/abs/2410.13509)  
**Date:** 2024-10-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.13509v1/

## Key points

- Large Language Models (LLMs) struggle with accurate responses due to hallucinations
- Retrieval-Augmented Generation (RAG) enhances LLMs by retrieving knowledge from external sources
- Instruction tuning optimizes LLMs to effectively utilize retrieved knowledge for diverse tasks
- Differentiable Data Rewards (DDR) aligns data preferences between different modules in RAG systems
- DDR outperforms supervised fine-tuning (SFT), especially for smaller-scale LLMs heavily relying on retrieved knowledge
- DDR improves the generation module's effectiveness in extracting key information and mitigating conflicts between parametric memory and external knowledge
- DDR focuses on aligning data preferences across different agents within a multi-agent system, avoiding overfitting training signals and enhancing RAG optimization compared to traditional methods like SFT

## Summary

, , , , 

In recent years, Large Language Models (LLMs) have shown remarkable capabilities in various natural language processing tasks. However, they often struggle with producing accurate responses due to hallucinations. To address this issue, Retrieval-Augmented Generation (RAG) has been introduced as a solution to enhance LLMs by retrieving knowledge from external sources. The integration of RAG pipelines into LLMs has been achieved through instruction tuning, which optimizes LLMs to effectively utilize retrieved knowledge for diverse tasks. While supervised fine-tuning (SFT) has been the conventional approach for adapting LLMs to RAG pipelines, it tends to lead to overfitting and overlooks the varying data preferences among agents within the system. In response to these limitations, a new method called Differentiable Data Rewards (DDR) is proposed in this paper. DDR aims to train RAG systems end-to-end by aligning data preferences between different modules. By collecting rewards and optimizing each agent with a rollout method, DDR prompts agents to sample potential responses, evaluate their impact on the overall system, and optimize outputs that improve system performance. Experimental results on knowledge-intensive tasks demonstrate that DDR outperforms SFT, especially for smaller-scale LLMs that heavily rely on retrieved knowledge. Additionally, DDR exhibits a stronger capability in aligning data preferences between RAG modules, making the generation module more effective in extracting key information from documents and mitigating conflicts between parametric memory and external knowledge. Previous research has focused on optimizing LLMs for RAG through methods like INFO-RAG and RA-DIT. While reinforcement learning algorithms have been used for preference optimization in individual agents, DDR stands out by focusing on aligning data preferences across different agents within the multi-agent system. This approach avoids overfitting training signals and enhances the effectiveness of RAG optimization compared to traditional methods like SFT. Overall, the DDR method presents a promising advancement in training RAG systems and improving the performance of LLMs when handling complex language generation tasks. The code for implementing DDR is available at https://github.com/OpenMatch/RAG-DDR.