# Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models

**Authors:** Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, Ji-Rong Wen  
**arXiv:** [2305.13112](https://arxiv.org/abs/2305.13112)  
**Date:** 2023-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.13112v1/

## Key points

- The paper explores the potential of large language models (LLMs) for developing conversational recommender systems (CRSs)
- The authors investigate the use of ChatGPT for conversational recommendation and identify limitations in the existing evaluation protocol
- They propose an interactive evaluation approach called iEvaLM that leverages LLM-based user simulators to address these limitations
- Experiments conducted on two publicly available CRS datasets demonstrate notable improvements compared to the prevailing evaluation protocol
- The importance of evaluating explainability in CRSs is highlighted, with ChatGPT exhibiting persuasive explanation generation for its recommendations
- The study provides a deeper understanding of the untapped potential of LLMs for CRSs and offers a more flexible and user-friendly evaluation framework

## Summary

The paper titled "Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models" by Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen explores the potential of large language models (LLMs) for developing more powerful conversational recommender systems (CRSs). These CRSs rely on natural language conversations to meet user needs. The authors specifically investigate the use of ChatGPT for conversational recommendation and identify limitations in the existing evaluation protocol. The current evaluation protocol places excessive emphasis on matching with ground-truth items or utterances generated by human annotators. This approach overlooks the interactive nature required for an effective CRS. To address this limitation, the authors propose an interactive evaluation approach called iEvaLM that leverages LLM-based user simulators. This approach enables simulation of various interaction scenarios between users and systems. Through experiments conducted on two publicly available CRS datasets, the authors demonstrate notable improvements compared to the prevailing evaluation protocol. Additionally, they highlight the importance of evaluating explainability in CRSs. ChatGPT exhibits persuasive explanation generation for its recommendations. Overall, this study provides a deeper understanding of the untapped potential of LLMs for CRSs and offers a more flexible and user-friendly evaluation framework for future research endeavors. The codes and data related to this work are publicly available at https://github.com/RUCAIBox/iEvaLM-CRS which can be used to further explore these topics in greater detail.