# The Power of Scale for Parameter-Efficient Prompt Tuning

**Authors:** Brian Lester, Rami Al-Rfou, Noah Constant  
**arXiv:** [2104.08691](https://arxiv.org/abs/2104.08691)  
**Date:** 2021-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.08691v1/

## Key points

- The paper explores the concept of prompt tuning to condition frozen language models for specific downstream tasks
- Their method involves learning "soft prompts" through backpropagation that can be tuned to incorporate signal from any number of labeled examples
- The end-to-end learned approach outperforms GPT-3's few-shot learning by a significant margin
- Prompt tuning becomes more competitive with scale, and matches the strong performance of model tuning as models exceed billions of parameters
- Prompt tuning is a simplification of the recently proposed "prefix tuning" by Li and Liang (2021), and the authors provide a comparison to this and other similar approaches
- Conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer compared to full model tuning.

## Summary

In their paper "The Power of Scale for Parameter-Efficient Prompt Tuning," Brian Lester, Rami Al-Rfou, and Noah Constant explore the concept of prompt tuning as a means to condition frozen language models for specific downstream tasks. Unlike GPT-3's discrete text prompts, their method involves learning "soft prompts" through backpropagation that can be tuned to incorporate signal from any number of labeled examples. The authors demonstrate that their end-to-end learned approach outperforms GPT-3's few-shot learning by a significant margin. Through ablations on model size using T5, the authors show that prompt tuning becomes more competitive with scale. As models exceed billions of parameters, their method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned). This finding is particularly relevant since large models are costly to share and serve, and reusing one frozen model for multiple downstream tasks can ease this burden. Prompt tuning can be seen as a simplification of the recently proposed "prefix tuning" by Li and Liang (2021), and the authors provide a comparison to this and other similar approaches. Additionally, they show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer compared to full model tuning. Overall, this paper presents an innovative approach to fine-tuning language models for specific tasks that outperforms existing methods while reducing computational costs associated with training large models from scratch.