# Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency

**Authors:** Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, Jie Zhou  
**arXiv:** [2106.02228](https://arxiv.org/abs/2106.02228)  
**Date:** 2021-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.02228v1/

## Key points

- The paper proposes a framework called AIH (Addressing Inquiries about History) to evaluate the consistency capacity of open-domain chatbots.
- Traditional methods of evaluating chatbot consistency are costly, inefficient, and prone to subjective bias.
- The AIH framework induces chatbots to redeclare historical facts or opinions in response to appropriate inquiries about the dialogue history.
- Conversations between chatbots are more efficient than human-bot interactions and help alleviate subjective bias.
- The framework enables efficient evaluation of chatbot consistency by obtaining a dialog session with high contradiction possibilities.
- Human judges or a natural language inference (NLI) model can be employed for contradiction recognition.
- Chatbots can be ranked according to their contradiction statistics based on this recognition.
- Experiments demonstrate that this approach efficiently and reliably assesses the consistency capacity of chatbots with high ranking correlation with human evaluation indicating its effectiveness.
- The authors have released the AIH framework with hopes of improving the consistency capacity of chatbots.

## Summary

The paper titled "Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency" proposes a framework called AIH (Addressing Inquiries about History) to evaluate the consistency capacity of open-domain chatbots. The consistency capacity refers to the ability of a chatbot to avoid presenting contradictory responses about facts or opinions during a conversation. Traditional methods of evaluating chatbot consistency are costly, inefficient, and prone to subjective bias. To address these challenges, the AIH framework induces chatbots to redeclare historical facts or opinions in response to appropriate inquiries about the dialogue history. This is achieved through conversations between chatbots which are more efficient than human-bot interactions and help alleviate subjective bias. By rapidly obtaining a dialog session that contains responses with high contradiction possibilities, the framework enables efficient evaluation of chatbot consistency. In the contradiction recognition stage, either human judges or a natural language inference (NLI) model can be employed to determine whether the answers provided by the chatbots contradict historical information. Based on this recognition, chatbots can be ranked according to their contradiction statistics. Experiments conducted on open-domain chatbots demonstrate that this approach efficiently and reliably assesses the consistency capacity of chatbots with high ranking correlation with human evaluation indicating its effectiveness. The authors have released the AIH framework with hopes of improving the consistency capacity of chatbots. Overall, this paper presents an innovative and practical solution for evaluating open-domain chatbot consistency using an efficient conversation-based approach.