# Redefining Retrieval Evaluation in the Era of LLMs

**Authors:** Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, Fabrizio Silvestri  
**arXiv:** [2510.21440](https://arxiv.org/abs/2510.21440)  
**Date:** 2025-10-24  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2510.21440v1/

## Key points

- Authors address limitations of traditional IR metrics in the context of Retrieval Augmented Generation (RAG) systems involving Large Language Models (LLMs)
- Conventional IR metrics like nDCG, MAP, and MRR do not consider negative impact of irrelevant documents on generation quality in RAG systems
- Authors propose a utility-based annotation schema to quantify positive contribution of relevant passages and negative impact of distracting ones
- Introduction of UDCG (Utility and Distraction-aware Cumulative Gain) metric designed to optimize correlation with end-to-end answer accuracy by incorporating an LLM-oriented positional discount
- Experimental results show UDCG significantly improves correlation by up to 36% compared to traditional metrics
- Proposed framework addresses misalignments between human vs. machine position discount and human relevance vs. machine utility for more accurate prediction of RAG performance

## Summary

In their paper titled "Redefining Retrieval Evaluation in the Era of LLMs," authors Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, and Fabrizio Silvestri address the limitations of traditional Information Retrieval (IR) metrics in the context of Retrieval Augmented Generation (RAG) systems. These systems involve Large Language Models (LLMs) that process all retrieved documents simultaneously. This is unlike humans who sequentially examine them. The authors highlight that conventional IR metrics like nDCG, MAP, and MRR do not consider the negative impact of irrelevant documents on generation quality in RAG systems. To overcome these challenges, the authors propose a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. Building on this foundation, they introduce UDCG (Utility and Distraction-aware Cumulative Gain), a metric designed to optimize correlation with end-to-end answer accuracy by incorporating an LLM-oriented positional discount. Experimental results on five datasets and six LLMs demonstrate that UDCG significantly improves correlation by up to 36% compared to traditional metrics. This work represents a critical step towards aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components. By addressing the misalignments between human vs. machine position discount and human relevance vs. machine utility, the proposed framework offers a more accurate prediction of RAG performance in modern information retrieval systems.