# How Many Data Points is a Prompt Worth?

**Authors:** Teven Le Scao, Alexander M. Rush  
**arXiv:** [2103.08493](https://arxiv.org/abs/2103.08493)  
**Date:** 2021-03-15  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2103.08493v2/

## Key points

- The paper explores the use of task-specific prompts versus generic model heads in fine-tuning pretrained models for classification.
- Proponents of prompting argue that it provides a method for injecting task-specific guidance which is beneficial in low-data regimes.
- The main benefit of prompting is data efficiency rather than compute efficiency.
- Rigorous testing was conducted to compare prompted and head-based fine-tuning in equal conditions across many tasks and data sizes.
- Prompting does indeed provide a benefit, and this benefit can be quantified per task.
- Results show that prompting is often worth hundreds of data points on average across classification tasks.
- The experiments were computationally intensive but carbon neutral due to running almost two thousand runs on a single Nvidia V100 GPU with each experiment taking under an hour.
- There are inherent risks associated with introducing human biases into models through prompts and repeating biases already present within the language model.
- Prompting mostly relies on the pretrained model in few-shot settings where human input is minimal.
- The paper provides valuable insights into the benefits and limitations of using task-specific prompts versus generic model heads in fine-tuning pretrained models for classification tasks.

## Summary

This paper explores the use of task-specific prompts versus generic model heads in fine-tuning pretrained models for classification. While proponents of prompting argue that it provides a method for injecting task-specific guidance which is beneficial in low-data regimes, the main benefit of prompting is data efficiency rather than compute efficiency. To quantify this benefit, the authors conducted rigorous testing of prompts in a fair setting by comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, they found that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth hundreds of data points on average across classification tasks. The experiments were computationally intensive but carbon neutral due to running almost two thousand runs on a single Nvidia V100 GPU with each experiment taking under an hour. The authors also highlighted the potential benefits of prompt completion in low-resource language applications. However, there are inherent risks associated with introducing human biases into models through prompts and repeating biases already present within the language model. Prompting mostly relies on the pretrained model in few-shot settings where human input is minimal. Overall, this paper provides valuable insights into the benefits and limitations of using task-specific prompts versus generic model heads in fine-tuning pretrained models for classification tasks.