# Finetuned Language Models Are Zero-Shot Learners

**Authors:** Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le  
**arXiv:** [2109.01652](https://arxiv.org/abs/2109.01652)  
**Date:** 2021-09-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.01652v1/

## Key points

- "Finetuned Language Models Are Zero-Shot Learners" explores a method for enhancing zero-shot learning capabilities of language models.
- Fine-tuning language models on diverse tasks described through instructions significantly improves performance on unseen tasks.
- Pretrained language model with 137 billion parameters is instruction-tuned on over 60 NLP tasks using templates.
- Resulting model, FLAN, outperforms unmodified counterpart and GPT-3 on 19 out of 25 evaluated tasks.
- FLAN also surpasses few-shot GPT-3 on benchmark datasets including ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.
- Number of tasks used for instruction tuning and scale of the language model are crucial factors contributing to success.
- Instruction tuning enhances zero-shot learning in language models and highlights importance of task diversity and model scale.

## Summary

This paper titled "Finetuned Language Models Are Zero-Shot Learners" explores a straightforward yet effective method for enhancing the zero-shot learning capabilities of language models. The authors demonstrate that by fine-tuning language models on a diverse range of tasks described through instructions, they can significantly improve the models' performance on unseen tasks. To conduct their study, the authors utilize a pretrained language model with 137 billion parameters and instruction-tune it on more than 60 natural language processing (NLP) tasks, which are verbalized using instruction templates. The resulting model, named FLAN, is then evaluated on task types that were not included in the training data. Remarkably, FLAN outperforms its unmodified counterpart and even surpasses the zero-shot performance of GPT-3, a larger 175 billion parameter model, on 19 out of 25 evaluated tasks. Furthermore, FLAN exhibits superior performance compared to few-shot GPT-3 by a significant margin on several benchmark datasets including ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA and StoryCloze. Ablation studies conducted by the authors reveal that both the number of tasks used for instruction tuning and the scale of the language model are crucial factors contributing to the success of this approach. In summary, this paper presents an innovative technique called instruction tuning that substantially enhances zero-shot learning in language models. The results demonstrate that FLAN outperforms both its unmodified counterpart and larger-scale models like GPT-3 in various NLP tasks. These findings highlight the importance of incorporating instructions during fine-tuning and emphasize the significance of task diversity and model scale in achieving improved zero-shot performance.