# Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

**Authors:** Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, Akiko Aizawa  
**arXiv:** [2011.01060](https://arxiv.org/abs/2011.01060)  
**Date:** 2020-11-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2011.01060v1/

## Key points

- Introduction of a new dataset called 2WikiMultiHopQA for Multi-hop Question Answering
- Dataset challenges reasoning and inference skills by requiring models to read multiple paragraphs
- Comprehensive evidence information provided in the dataset enhances transparency and evaluation of model performance
- Question-answer pairs are carefully designed to ensure multi-hop reasoning steps are necessary
- Logical rules create natural yet challenging questions that demand multi-hop reasoning abilities
- Structured data from Wikidata guarantees realistic questions that require advanced reasoning capabilities
- Tasks include answer prediction, SFs prediction, and evidence generation with evaluation metrics like EM and F1 score
- Joint metrics introduced to evaluate overall model capacity by combining precision and recall for various aspects
- Four types of questions included: comparison, inference, compositional, and bridge comparison
- Post-processing techniques applied for data quality assurance including balancing yes/no questions and eliminating ambiguous cases
- Distractor paragraphs collected using bigram tf-idf similarity measures to provide context for each question
- Dataset statistics established through benchmark setting using single-hop model for train-dev-test splits with five-fold cross-validation.

## Summary

The researchers introduce a new dataset called 2WikiMultiHopQA for Multi-hop Question Answering that utilizes both structured and unstructured data from Wikipedia and Wikidata. The dataset aims to challenge reasoning and inference skills by requiring models to read multiple paragraphs in order to answer complex questions. Unlike existing datasets, 2WikiMultiHopQA provides comprehensive evidence information that outlines the reasoning path from the question to the answer, enhancing transparency and evaluation of model performance. The generation of question-answer pairs in the dataset is carefully designed using a pipeline and templates that ensure multi-hop reasoning steps are necessary for answering each question. Logical rules are employed to create natural yet challenging questions that demand multi-hop reasoning abilities from models. By leveraging structured data from Wikidata, the researchers guarantee that questions are not only realistic but also require advanced reasoning capabilities. The tasks formulated in this study include answer prediction, sentence-level supporting facts (SFs) prediction, and evidence generation. Evaluation metrics such as exact match (EM) and F1 score are used to assess model performance across these tasks. Joint metrics are introduced to evaluate the overall capacity of models by combining precision and recall for answer spans, SFs, and evidence. Furthermore, the dataset includes four types of questions: comparison, inference, compositional, and bridge comparison. Bridge questions involve a bridge entity connecting two paragraphs and represent a subtype of inference and compositional questions. To ensure data quality, post-processing techniques were applied to balance yes/no questions and eliminate ambiguous cases. Additionally, distractor paragraphs were collected using bigram tf-idf similarity measures to provide context for each question. Dataset statistics were established through a benchmark setting using a single-hop model for train-dev-test splits with five-fold cross-validation. Overall, this study contributes a valuable resource for evaluating multi-hop QA models by providing a challenging dataset with detailed explanations for predictions and rigorous evaluation criteria. The full dataset along with baseline models can be accessed on GitHub for further research purposes.