# Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

**Authors:** Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, Chaowei Xiao  
**arXiv:** [2209.07511](https://arxiv.org/abs/2209.07511)  
**Date:** 2022-09-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.07511v1/

## Key points

- The paper proposes a method called Test-Time Prompt Tuning (TPT) to improve the zero-shot generalization capability of pre-trained vision-language models like CLIP.
- TPT can learn adaptive prompts on the fly with a single test sample, which addresses the limitation of training on domain-specific data and improves model's ability to generalize to unseen domains.
- TPT optimizes the prompt by minimizing entropy with confidence selection for image classification tasks, so that the model has consistent predictions across different augmented views of each test sample.
- TPT outperforms previous prompt tuning approaches that require additional task-specific training data in two scenarios: natural distribution shifts and cross-dataset generalization with unseen categories.
- TPT improves CLIP's zero-shot top-1 accuracy by 3.6% on average when evaluating generalization to natural distribution shifts.
- TPT performs on par with state-of-the-art approaches that use additional training data when evaluating cross dataset generalization with unseen categories.
- The proposed method has significant implications for real world applications where models need to generalize to new domains without access to large amounts of labeled data.
- Interested readers can find more information about the work and access code implementations on the authors' project page.

## Summary

The paper titled "Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models" proposes a novel method to improve the zero-shot generalization capability of pre-trained vision-language models like CLIP. While previous works have shown promising results by learning prompts from downstream task data, such training on domain-specific data limits the model's ability to generalize to unseen domains. To address this limitation, the authors propose Test-Time Prompt Tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. For image classification tasks, TPT optimizes the prompt by minimizing entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. The authors evaluate TPT's performance in two scenarios: natural distribution shifts and cross-dataset generalization with unseen categories. In both cases, TPT outperforms previous prompt tuning approaches that require additional task-specific training data. In particular, TPT improves CLIP's zero-shot top-1 accuracy by 3.6% on average when evaluating generalization to natural distribution shifts. Additionally, TPT performs on par with state-of-the-art approaches that use additional training data when evaluating cross dataset generalization with unseen categories. The proposed method has significant implications for real world applications where models need to generalize to new domains without access to large amounts of labeled data. The authors provide a project page where interested readers can find more information about their work and access code implementations. Overall, this paper presents an innovative approach that could significantly enhance the practical utility of pre trained vision language models in various downstream tasks.