# TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

**Authors:** Thomas Wolf, Victor Sanh, Julien Chaumond, Clement Delangue  
**arXiv:** [1901.08149](https://arxiv.org/abs/1901.08149)  
**Date:** 2019-01-23  
**Summary page:** https://www.summarizepaper.com/en/paper/1901.08149v1/

## Key points

- TransferTransfo is a novel approach to generative data-driven dialogue systems
- Developed by Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue
- Combines transfer learning with a high-capacity Transformer model
- Utilizes a multi-task objective during fine-tuning
- Outperforms existing conversational models like memory augmented seq2seq and information retrieval models
- Evaluated using the PERSONA-CHAT dataset from Conversational Intelligence Challenge 2
- Achieves significant improvements in perplexity, Hits@1, and F1 score compared to previous approaches
- Enhances language understanding, response relevance, and overall conversational quality
- Offers a promising solution for improving chatbots and dialogue systems.

## Summary

TransferTransfo is a novel approach to generative data-driven dialogue systems, such as chatbots, developed by Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. This approach combines transfer learning with a high-capacity Transformer model to achieve significant improvements over existing conversational models. The TransferTransfo model utilizes a multi-task objective during fine-tuning, which involves combining multiple unsupervised prediction tasks. This training scheme allows the model to learn from various sources of data and improve its performance in generating coherent and contextually relevant responses. In comparison to state-of-the-art end-to-end conversational models like memory augmented seq2seq and information retrieval models, TransferTransfo demonstrates superior performance. The privately held PERSONA-CHAT dataset from the Conversational Intelligence Challenge 2 was used to evaluate the approach. The results obtained by TransferTransfo on this dataset are remarkable. The fine-tuned model achieves a perplexity of 16.28 (a 45% absolute improvement), Hits@1 of 80.7 (a 46% absolute improvement), and an F1 score of 19.5 (a 20% absolute improvement). These metrics indicate that TransferTransfo outperforms previous approaches in terms of language understanding, response relevance, and overall conversational quality. Overall, TransferTransfo presents a promising solution for enhancing the capabilities of chatbots and other dialogue systems. Its combination of transfer learning and Transformer architecture enables it to generate more accurate and contextually appropriate responses, leading to improved user experiences in conversational interactions.