# On (Normalised) Discounted Cumulative Gain as an Offline Evaluation Metric for Top-$n$ Recommendation

**Authors:** Olivier Jeunen, Ivan Potapov, Aleksei Ustimenko  
**arXiv:** [2307.15053](https://arxiv.org/abs/2307.15053)  
**Date:** 2023-07-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.15053v1/

## Key points

- Fundamental difference between online and offline evaluation metrics highlighted
- Proposal to use open-source simulation environments to overcome need for online ground truth result
- Study by Olivier Jeunen, Ivan Potapov, and Aleksei Ustimenko focuses on bridging gap between offline and online paradigms through critical examination of (n)DCG metric
- Research formalizes problem setting within session-based feed recommendation framework with contextual features encoding user trajectories
- Two-stage ranking setup commonly employed due to vast item catalogue size in real-world systems for personalized recommendations
- In-depth analysis of (n)DCG as an unbiased estimator of online reward, highlighting deviations from traditional usage in Information Retrieval
- Normalizing the metric can lead to inconsistencies in ranking competing methods by their relative performance
- Correlation analyses show unbiased DCG estimates strongly correlate with online rewards even when assumptions are violated, but not true for normalized DCG variants
- Work contributes valuable insights into refining offline evaluation metrics like (n)DCG to better approximate online experiment outcomes

## Summary

In recent research, a fundamental difference has been highlighted between online evaluation metrics and offline evaluation metrics. This discrepancy becomes especially pronounced when considering reinforcement learning methods and interpreting organic implicit feedback as a user preference signal. The use of open-source simulation environments has been proposed to overcome the need for an online ground truth result. However, the question remains whether conclusions drawn from simulation results accurately reflect those from real-world experiments. In this study by Olivier Jeunen, Ivan Potapov, and Aleksei Ustimenko, the focus is on bridging the gap between offline and online paradigms through a critical examination of the widely used (n)DCG metric. The aim is to develop offline evaluation methodologies that closely mirror the outcomes of online experiments. The research delves into formalizing the problem setting within a session-based feed recommendation framework where user trajectories on a platform are encoded with contextual features. Due to the vast item catalogue size in real-world systems, a two-stage ranking setup is commonly employed to efficiently score and rank items for personalized recommendations. The study presents an in-depth analysis of (n)DCG as an unbiased estimator of online reward and highlights deviations from its traditional usage in Information Retrieval. It is demonstrated that normalizing the metric can lead to inconsistencies in ranking competing methods by their relative performance. Through correlation analyses between offline and online experiments conducted on a large-scale recommendation platform, it is shown that unbiased DCG estimates strongly correlate with online rewards even when certain assumptions are violated. However, this correlation does not hold true for normalized DCG variants, suggesting limitations in practical utility. Overall, this work contributes valuable insights into refining offline evaluation metrics like (n)DCG to better approximate online experiment outcomes and enhance the efficacy of recommendation systems across different evaluation paradigms.