# MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

**Authors:** Yixuan Tang, Yi Yang  
**arXiv:** [2401.15391](https://arxiv.org/abs/2401.15391)  
**Date:** 2024-01-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.15391v1/

## Key points

- Authors address limitations of existing RAG methods in answering multi-hop queries
- Introduce a novel dataset called MultiHop-RAG consisting of knowledge base, multi-hop queries, ground-truth answers, and supporting evidence
- Conduct two experiments: comparing different embedding models for retrieving evidence and examining capabilities of various language models in reasoning and answering multi-hop queries
- Existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries
- Categorize multi-hop queries into four types: Inference query, Comparison query, Temporal query, and Null query
- Construct RAG knowledge base by extracting factual sentences from news articles and generating claims using GPT-4 with disambiguated topics and entities as bridges
- MultiHop-RAG provides a resource for developing effective RAG systems and evaluating language models' reasoning capabilities
- Dataset and implemented RAG system are publicly available for the community to access.

## Summary

, , , , 

The authors of this paper address the limitations of existing  in answering multi-hop queries. These types of queries require retrieving and reasoning over multiple pieces of supporting evidence, which current RAG methods struggle with. To overcome this challenge, the authors introduce a novel dataset called . This dataset consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and associated supporting evidence. The knowledge base is constructed using an English news article dataset. To evaluate the benchmarking utility of MultiHop-RAG, the authors conduct two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. The second experiment examines the capabilities of various state-of-the-art language models in reasoning and answering multi-hop queries given the evidence. The results from both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. To better understand these types of queries, the authors categorize them into four types: Inference query, Comparison query, Temporal query, and Null query. Each type requires retrieving and analyzing evidence from multiple sources to infer relationships, compare data points, sequence events over time or determine if an answer cannot be derived from the knowledge base. To construct the RAG knowledge base, factual sentences are extracted from news articles and inputted into GPT-4 to generate claims that are clarified with disambiguated topics and entities acting as bridges for constructing multi-hop queries. Specific multi-hop queries related to the same bridge-topic or bridge-entity are generated along with their correct answers. In addition to providing a valuable resource for developing effective RAG systems, MultiHop-RAG also allows for the evaluation of language models' reasoning capabilities. By comparing their responses with the ground truth answers of queries requiring reasoning over multiple retrieved chunks, researchers can better understand and improve these models. Overall, MultiHop-RAG is a crucial step towards enhancing the adoption of large language models in practical applications. The dataset and implemented RAG system are publicly available for the community to access and utilize.