Results of the summarizing process for the arXiv paper: 2510.21440v1

In their paper titled "Redefining Retrieval Evaluation in the Era of LLMs," authors Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, and Fabrizio Silvestri address the limitations of traditional Information Retrieval (IR) metrics in the context of Retrieval Augmented Generation (RAG) systems. These systems involve Large Language Models (LLMs) that process all retrieved documents simultaneously. This is unlike humans who sequentially examine them. The authors highlight that conventional IR metrics like nDCG, MAP, and MRR do not consider the negative impact of irrelevant documents on generation quality in RAG systems. To overcome these challenges, the authors propose a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. Building on this foundation, they introduce UDCG (Utility and Distraction-aware Cumulative Gain), a metric designed to optimize correlation with end-to-end answer accuracy by incorporating an LLM-oriented positional discount. Experimental results on five datasets and six LLMs demonstrate that UDCG significantly improves correlation by up to 36% compared to traditional metrics. This work represents a critical step towards aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components. By addressing the misalignments between human vs. machine position discount and human relevance vs. machine utility, the proposed framework offers a more accurate prediction of RAG performance in modern information retrieval systems.

• Authors address limitations of traditional IR metrics in the context of Retrieval Augmented Generation (RAG) systems involving Large Language Models (LLMs)
• Conventional IR metrics like nDCG, MAP, and MRR do not consider negative impact of irrelevant documents on generation quality in RAG systems
• Authors propose a utility-based annotation schema to quantify positive contribution of relevant passages and negative impact of distracting ones
• Introduction of UDCG (Utility and Distraction-aware Cumulative Gain) metric designed to optimize correlation with end-to-end answer accuracy by incorporating an LLM-oriented positional discount
• Experimental results show UDCG significantly improves correlation by up to 36% compared to traditional metrics
• Proposed framework addresses misalignments between human vs. machine position discount and human relevance vs. machine utility for more accurate prediction of RAG performance

Redefining Retrieval Evaluation in the Era of LLMs

In recent years, there has been a significant shift towards using Large Language Models (LLMs) in information retrieval systems. These models have shown great potential in improving the accuracy and efficiency of document retrieval by processing all retrieved documents simultaneously, unlike traditional methods that rely on sequential examination by humans. However, this advancement also brings about new challenges in evaluating the performance of these systems.

In their paper titled "Redefining Retrieval Evaluation in the Era of LLMs," authors Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, and Fabrizio Silvestri address these limitations and propose a new evaluation framework for Retrieval Augmented Generation (RAG) systems. This research is crucial as it enables more reliable assessment of RAG components and aligns IR evaluation with LLM consumers.

The Limitations of Traditional Information Retrieval Metrics

Traditional metrics used to evaluate information retrieval systems such as nDCG (Normalized Discounted Cumulative Gain), MAP (Mean Average Precision), and MRR (Mean Reciprocal Rank) were designed for human consumption. They do not consider the negative impact of irrelevant documents on generation quality in RAG systems where LLMs process all retrieved documents simultaneously.

This misalignment between human vs. machine position discount and human relevance vs. machine utility can lead to inaccurate evaluations of RAG system performance. For instance, a document that is relevant but appears at a lower rank may be discounted by traditional metrics even though it contributes positively to the overall answer accuracy.

A New Utility-Based Annotation Schema

To overcome these challenges, the authors propose a utility-based annotation schema that quantifies both the positive contribution of relevant passages and the negative impact of distracting ones. This approach takes into account both relevance and utility when evaluating the performance of RAG systems.

The utility-based annotation schema assigns a score to each document based on its relevance and utility. Relevant documents receive a high score, while irrelevant or distracting ones receive a low score. This allows for a more accurate assessment of the impact of each document on the overall answer accuracy.

Introducing UDCG - A New Metric for Evaluating RAG Systems

Building on this foundation, the authors introduce UDCG (Utility and Distraction-aware Cumulative Gain), a metric designed specifically for RAG systems. UDCG incorporates an LLM-oriented positional discount that takes into account the simultaneous processing of all retrieved documents by LLMs.

This new metric aims to optimize correlation with end-to-end answer accuracy in RAG systems. It considers both relevance and utility when calculating cumulative gain, providing a more accurate measure of system performance compared to traditional metrics.

Experimental Results

To validate their proposed framework, the authors conducted experiments on five datasets and six LLMs. The results showed that UDCG significantly improves correlation with end-to-end answer accuracy by up to 36% compared to traditional metrics like nDCG and MAP.

These findings demonstrate the effectiveness of UDCG in accurately evaluating RAG systems' performance in modern information retrieval settings where LLMs are used extensively.

The Importance of Redefining Retrieval Evaluation

The research presented in this paper is crucial as it addresses the limitations of traditional IR metrics in evaluating modern information retrieval systems that utilize LLMs. By introducing a new utility-based annotation schema and metric, it enables more reliable assessment of RAG components and aligns IR evaluation with LLM consumers.

Furthermore, this work represents an important step towards improving our understanding of how these advanced models process information and contribute to overall system performance. It also highlights the need for continued research in developing evaluation frameworks that accurately reflect the capabilities and limitations of LLMs in information retrieval.

Conclusion

In conclusion, the paper "Redefining Retrieval Evaluation in the Era of LLMs" by Trappolini et al. presents a new utility-based annotation schema and metric, UDCG, for evaluating RAG systems. This research addresses the misalignments between human vs. machine position discount and human relevance vs. machine utility in traditional IR metrics and provides a more accurate measure of system performance.

The experimental results demonstrate the effectiveness of UDCG in improving correlation with end-to-end answer accuracy compared to traditional metrics. This work represents an important step towards aligning IR evaluation with LLM consumers and enables more reliable assessment of RAG components in modern information retrieval systems.