# Towards a Human-like Open-Domain Chatbot

**Authors:** Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le  
**arXiv:** [2001.09977](https://arxiv.org/abs/2001.09977)  
**Date:** 2020-01-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2001.09977v1/

## Key points

- Authors present Meena, a multi-turn open-domain chatbot trained on social media conversations
- Meena's neural network has 2.6 billion parameters and is trained to minimize perplexity
- Authors propose Sensibleness and Specificity Average (SSA) as a human evaluation metric for conversation quality
- Strong correlation found between perplexity and SSA scores
- Optimizing perplexity may lead to achieving a human-level SSA score of 86%
- Best performing version of Meena achieved an SSA score of 72% in multi-turn evaluation
- Full version of Meena includes additional features like filtering mechanism and tuned decoding
- Full version of Meena scored an impressive SSA score of 79%, 23% higher than the next highest scoring chatbot evaluated by the authors
- Research demonstrates significant progress in developing a human-like open domain chatbot

## Summary

In their paper titled "Towards a Human-like Open-Domain Chatbot," authors Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu and Quoc V. Le present Meena - a multi-turn open-domain chatbot that is trained end-to-end on data mined and filtered from public domain social media conversations. The neural network used for Meena has 2.6 billion parameters and is trained to minimize perplexity - an automatic metric that measures the quality of multi-turn conversations. To evaluate Meena's performance in terms of conversation quality more accurately than perplexity alone can provide, the authors propose a human evaluation metric called Sensibleness and Specificity Average (SSA). The experiments conducted by the authors reveal a strong correlation between perplexity and SSA scores. This suggests that if perplexity can be further optimized it may be possible to achieve a human level SSA score of 86%. Notably the best performing version of Meena trained with low perplexity scores high on SSA with a multi-turn evaluation score of 72%. Furthermore the full version of Meena includes additional features such as a filtering mechanism and tuned decoding which enhances its performance significantly resulting in an impressive SSA score of 79%, 23% higher than the next highest scoring chatbot evaluated by the authors. Overall this research demonstrates significant progress in developing a human like open domain chatbot. By training Meena to minimize perplexity and optimizing its performance based on SSA evaluations there is potential to create chatbots that can engage in conversations at a level comparable to human interactions.