# BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

**Authors:** Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi  
**arXiv:** [2301.12597](https://arxiv.org/abs/2301.12597)  
**Date:** 2023-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.12597v1/

## Key points

- BLIP-2 is a new approach in vision-and-language pre-training that addresses the cost barrier by leveraging pre-trained image encoders and frozen language models.
- BLIP-2 uses a lightweight Querying Transformer in two distinct stages: bootstrapping vision-language representation learning from a frozen image encoder and transitioning to vision-to-language generative learning from a frozen language model.
- Despite having fewer trainable parameters, BLIP-2 achieves state-of-the-art performance in various vision-language tasks, surpassing Flamingo80B by 8.7% in zero-shot VQAv2 while using significantly fewer parameters.
- The framework of BLIP-2 includes a two-stage strategy with a Querying Transformer at its core, effectively bridging the modality gap between vision and language.
- BLIP-2 emphasizes the importance of cross-modal alignment when leveraging pre-trained unimodal models for Vision-Language Processing (VLP), showcasing capabilities in zero-shot image-to-text generation following natural language instructions.

## Summary

In the realm of vision-and-language pre-training, the cost has been a significant barrier due to the training of large-scale models end-to-end. To address this issue, a new approach called BLIP-2 has been introduced. BLIP-2 is a versatile and efficient pre-training strategy that leverages pre-trained image encoders and large language models that are already frozen. This method bridges the gap between vision and language using a lightweight Querying Transformer, which undergoes pre-training in two distinct stages. The first stage of BLIP-2 focuses on bootstrapping vision-language representation learning from a frozen image encoder. Subsequently, in the second stage, the model transitions to vision-to-language generative learning from a frozen language model. Despite having significantly fewer trainable parameters compared to existing methods, BLIP-2 manages to achieve state-of-the-art performance across various vision-language tasks. For instance, BLIP-2 surpasses Flamingo80B by 8.7% in zero-shot VQAv2 while utilizing 54 times fewer trainable parameters. Additionally, this innovative approach showcases emerging capabilities in zero-shot image-to-text generation that can effectively follow natural language instructions. Moreover, the framework of BLIP-2 is illustrated in Figure 1, providing an overview of its methodology. By employing a two-stage strategy with a Querying Transformer at its core, BLIP-2 successfully bridges the modality gap between vision and language. The utilization of pre-trained vision models for high-quality visual representation coupled with the strengths of large language models (LLMs) for robust language generation and zero-shot transfer abilities further enhances the efficiency and effectiveness of this approach. One key aspect highlighted in this paper is the importance of facilitating cross-modal alignment when leveraging pre-trained unimodal models for Vision-Language Processing (VLP). While freezing unimodal models during pre-training helps reduce computation costs and mitigate catastrophic forgetting issues, aligning visions with languages poses unique challenges—especially considering that LLMs have not been exposed to images during their initial training phase. Overall, BLIP-2 presents a promising solution to the challenges associated with vision-and-language pre-training by offering a cost-effective and computationally efficient approach that delivers impressive performance results across various tasks within this domain.