# Learning to Prompt for Vision-Language Models

**Authors:** Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu  
**arXiv:** [2109.01134](https://arxiv.org/abs/2109.01134)  
**Date:** 2021-09-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.01134v1/

## Key points

- Vision-language pre-training as a promising method for representation learning
- Advantages of vision-language pre-training: broader source of supervision, ability to transfer knowledge through zero-shot learning
- Major challenge in deploying vision-language models: prompt engineering
- Prompt engineering requires domain expertise and significant time investment
- Slight changes in the prompt can have a significant impact on performance
- Different downstream tasks require specific prompt designs, complicating deployment process
- Proposed solution: context optimization (CoOp)
- CoOp models context in prompts using continuous representations and performs end-to-end learning while keeping pre-trained parameters fixed
- CoOp enables fully automated design of task-relevant prompts
- Experiments on 11 datasets show that CoOp transforms pre-trained vision-language models into data-efficient visual learners
- CoOp outperforms handcrafted prompts with as few as one or two shots, achieving significant improvements with more shots (e.g., 16 shots)
- CoOp exhibits strong robustness to distribution shift during deployment
- CoOp automates the prompt engineering process in vision-language models, enabling efficient deployment and impressive performance gains compared to handcrafted prompts across various datasets.

## Summary

In recent years, vision-language pre-training has gained attention as a promising method for representation learning. Unlike traditional approaches that rely on images and discrete labels, vision-language pre-training aligns images and raw text using separate encoders. This paradigm offers several advantages, including a broader source of supervision and the ability to transfer knowledge to downstream tasks through zero-shot learning. However, one major challenge in deploying these models is prompt engineering. Designing an appropriate prompt, especially for context words surrounding a class name, requires domain expertise and significant time investment for fine-tuning the wording. Even slight changes in the prompt can have a significant impact on performance. Additionally, different downstream tasks often require specific prompt designs, further complicating the deployment process. To address this challenge, the authors propose a novel approach called context optimization (CoOp). The main idea behind CoOp is to model context in prompts using continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters fixed. This allows for fully automated design of task-relevant prompts. The authors conducted experiments on 11 datasets to evaluate the effectiveness of CoOp. The results show that CoOp successfully transforms pre-trained vision-language models into data-efficient visual learners. With as few as one or two shots, CoOp outperforms handcrafted prompts by a decent margin. Moreover, when using more shots (e.g., 16 shots), CoOp achieves significant improvements with an average gain of around 17% and reaching over 50% in some cases. Another notable advantage of CoOp is its strong robustness to distribution shift which means that even when faced with changes in data distribution during deployment, CoOp continues to perform well. Overall, this paper introduces an innovative solution to the challenge of prompt engineering in vision-language models by automating the design process through context optimization (CoOp). This enables efficient deployment and achieves impressive performance gains compared to handcrafted prompts across various datasets.