# Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

**Authors:** Andrea Madotto, Zihan Liu, Zhaojiang Lin, Pascale Fung  
**arXiv:** [2008.06239](https://arxiv.org/abs/2008.06239)  
**Date:** 2020-08-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2008.06239v2/

## Key points

- The paper addresses the challenge of learning task-oriented dialogue systems with limited data.
- Task-oriented dialogue systems rely on four interconnected modules: Natural Language Understanding (NLU), Dialogue State Tracking (DST), Dialogue Policy (DP), and Natural Language Generation (NLG).
- Collecting sufficient data for each module can be costly.
- The authors explore the use of language models as few-shot learners to overcome this challenge.
- Language models like GPT-2 and GPT-3 allow few-shot learning by priming the model with just a few examples.
- The authors evaluate the priming few-shot ability of these language models in NLU, DST, DP, and NLG tasks.
- Potential limitations include domain adaptation issues when transferring knowledge from one domain to another and potential overfitting when using too many examples to prime the model.
- Possible future directions for research include exploring different ways to combine multiple language models or incorporating external knowledge sources into the training process.

## Summary

In their paper "Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems," authors Andrea Madotto, Zihan Liu, Zhaojiang Lin, and Pascale Fung address the challenge of learning task-oriented dialogue systems with limited data. These systems rely on four interconnected modules: Natural Language Understanding (NLU), Dialogue State Tracking (DST), Dialogue Policy (DP), and Natural Language Generation (NLG). However, collecting sufficient data for each module can be costly. To overcome this challenge, the authors explore the use of language models as few-shot learners. Transfer learning is a common technique used to fine-tune large language models pre-trained on text or task-specific data with few samples. However, this approach requires fine-tuning steps and a set of parameters for each task. In contrast, language models like GPT-2 and GPT-3 allow few-shot learning by priming the model with just a few examples. The authors evaluate the priming few-shot ability of these language models in NLU, DST, DP, and NLG tasks. They also discuss potential limitations such as domain adaptation issues when transferring knowledge from one domain to another and potential overfitting when using too many examples to prime the model. The authors also highlight possible future directions for research such as exploring different ways to combine multiple language models or incorporating external knowledge sources into the training process. Overall, this paper provides insights into how language models can be leveraged as few-shot learners in task-oriented dialogue systems while also highlighting current limitations and potential areas for improvement in future work.