# Benchmarking Retrieval-Augmented Generation for Medicine

**Authors:** Guangzhi Xiong, Qiao Jin, Zhiyong Lu, Aidong Zhang  
**arXiv:** [2402.13178](https://arxiv.org/abs/2402.13178)  
**Date:** 2024-02-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.13178v1/

## Key points

- Authors Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang address challenges faced by large language models (LLMs) in medical question answering (QA)
- LLMs struggle with issues like hallucinations and outdated knowledge despite achieving state-of-the-art performance
- Proposal of retrieval-augmented generation (RAG) as a promising solution for improving LLM performance in medical QA
- Introduction of Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE) benchmark comprising 7,663 questions from five medical QA datasets
- Conducted extensive experiments using over 1.8 trillion prompt tokens across 41 combinations of various corpora, retrievers, and backbone LLMs with the MedRAG toolkit
- Significant accuracy improvements observed for six different LLMs compared to traditional prompting methods
- Enhanced performance elevates models like GPT-3.5 and Mixtral to levels similar to GPT-4
- Strategic combinations of diverse medical corpora and retrievers crucial for optimal RAG system performance
- Uncovered insights include log-linear scaling property and "lost-in-the-middle" effects within medical RAG systems
- Findings provide practical guidelines for implementing RAG systems tailored for medical applications
- Evaluation serves as a pivotal resource for advancing retrieval-augmented generation in medicine and highlights the importance of ongoing research for optimizing system performance

## Summary

In their paper titled "Benchmarking Retrieval-Augmented Generation for Medicine," authors Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang delve into the challenges faced by large language models (LLMs) in the realm of medical question answering (QA). Despite achieving state-of-the-art performance, LLMs still struggle with issues like hallucinations and outdated knowledge. To address these challenges, the authors propose the use of retrieval-augmented generation (RAG) as a promising solution that has gained widespread adoption. However, implementing a RAG system involves multiple flexible components, leading to a lack of established best practices for optimizing RAG settings for different medical purposes. In response to this gap in knowledge, the authors introduce the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a groundbreaking benchmark comprising 7,663 questions sourced from five medical QA datasets. Through MIRAGE, they conduct extensive experiments utilizing over 1.8 trillion prompt tokens across 41 combinations of various corpora, retrievers, and backbone LLMs using the MedRAG toolkit developed in their work. The results of their study demonstrate significant improvements in accuracy for six different LLMs when compared to traditional chain-of-thought prompting methods. Notably, these enhancements elevate the performance of models like GPT-3.5 and Mixtral to levels akin to GPT-4. The authors highlight that optimal performance is achieved through strategic combinations of diverse medical corpora and retrievers. Moreover, their research uncovers intriguing insights such as a log-linear scaling property and the "lost-in-the-middle" effects within medical RAG systems. These findings contribute valuable practical guidelines for implementing RAG systems tailored specifically for medical applications. Overall, this comprehensive evaluation serves as a pivotal resource for advancing the field of retrieval-augmented generation in medicine and underscores the importance of continued research in optimizing these systems for enhanced performance and accuracy.