# LiT: Zero-Shot Transfer with Locked-image Text Tuning

**Authors:** Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, Lucas Beyer  
**arXiv:** [2111.07991](https://arxiv.org/abs/2111.07991)  
**Date:** 2021-11-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.07991v1/

## Key points

- The paper introduces a novel method called contrastive-tuning for aligning image and text models
- The approach, called "Locked-image Text tuning" (LiT-tuning), combines locked pre-trained image models with unlocked text models
- LiT-tuning enables zero-shot transfer on various vision tasks such as image classification and retrieval
- The technique works reliably across different pre-training methods and architectures including ResNet, Vision Transformers, and MLP-Mixer
- Impressive results are achieved on three distinct image-text datasets, particularly with the ViT-g/14 model
- Notably, 84.5% zero-shot transfer accuracy is achieved on the ImageNet test set and 81.1% on the ObjectNet test set
- The paper presents an effective method for aligning image and text models through contrastive training while leveraging their pre-training capabilities
- The LiT-tuned model demonstrates remarkable performance in zero-shot transfer learning across various vision tasks

## Summary

, , , , 

The paper "LiT: Zero-Shot Transfer with Locked-image Text Tuning" introduces a novel method, called contrastive-tuning, for aligning image and text models through contrastive training while leveraging their pre-training capabilities. This approach, referred to as "Locked-image Text tuning" (LiT-tuning), combines locked pre-trained image models with unlocked text models to achieve the best results. By employing LiT-tuning, the model gains the ability to perform zero-shot transfer on various vision tasks such as image classification and retrieval. The proposed technique is versatile and works reliably across different pre-training methods and architectures including ResNet, Vision Transformers, and MLP-Mixer. The authors validate their approach using three distinct image-text datasets and achieve impressive results when applying it to the transformer-based ViT-g/14 model. Notably, they achieve 84.5% zero-shot transfer accuracy on the ImageNet test set and 81.1% on the challenging out-of-distribution ObjectNet test set. Overall, this paper presents an effective method for aligning image and text models through contrastive training while capitalizing on their pre-training capabilities. The LiT-tuned model demonstrates remarkable performance in zero-shot transfer learning across various vision tasks, showcasing its broad applicability and potential impact in the field of computer vision.