# Synthetic Test Collections for Retrieval Evaluation

**Authors:** Hossein A. Rahmani, Nick Craswell, Emine Yilmaz, Bhaskar Mitra, Daniel Campos  
**arXiv:** [2405.07767](https://arxiv.org/abs/2405.07767)  
**Date:** 2024-05-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.07767v1/

## Key points

- Test collections are crucial for evaluating information retrieval (IR) systems
- Large Language Models (LLMs) are increasingly used to generate synthetic datasets for various applications
- Previous research has primarily focused on using LLMs to create synthetic queries or documents, but their use in constructing fully synthetic test collections is a new area of exploration
- Initial findings suggest that synthetic test collections generated by LLMs can be reliable for evaluating IR systems
- It is important to explore and address potential biases that may arise from generating fully synthetic test collections
- Detailed experimental results comparing real human judgments with synthetic judgments generated by LLMs like GPT-4 and T5 show promising performance metrics such as NDCG@10 scores and Kendall correlations

## Summary

, , , , 

Test collections are essential for evaluating information retrieval (IR) systems, but obtaining diverse user queries and relevance judgments can be challenging and resource-intensive. Recently, there has been a growing interest in using Large Language Models (LLMs) to generate synthetic datasets for various applications. While previous research has focused on using LLMs to create synthetic queries or documents to enhance ranking models, the use of LLMs for constructing fully synthetic test collections is relatively unexplored. In this study, the researchers thoroughly investigate the feasibility of using LLMs to generate not only synthetic judgments but also synthetic queries for constructing test collections. The goal is to determine if these synthetic test collections can be reliable for evaluating IR systems and to assess any potential biases towards LLM-based models. Initial findings suggest that it is indeed possible to construct synthetic test collections using LLMs that are suitable for retrieval evaluation. Furthermore, exploring the potential biases that may arise from generating fully synthetic test collections is crucial for ensuring their quality, fairness, and reliability. While early analysis indicates minimal bias towards systems based on the same LLM used for generation, further research is needed to deepen our understanding of potential biases and develop strategies to mitigate them effectively. The researchers provide detailed experimental results comparing real human judgments with synthetic judgments generated by different LLMs like GPT-4 and T5, showing promising performance metrics such as NDCG@10 scores and Kendall correlations. Overall, this research contributes valuable insights into the use of LLMs for creating synthetic test collections in IR evaluation. By addressing potential biases and demonstrating the reliability of these collections, the study paves the way for future advancements in utilizing synthetic data generation techniques for enhancing information retrieval systems.