# F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

**Authors:** Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, Anelia Angelova  
**arXiv:** [2209.15639](https://arxiv.org/abs/2209.15639)  
**Date:** 2022-09-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.15639v2/

## Key points

- F-VLM is a novel object detection method that leverages Frozen Vision and Language Models (VLM)
- Simplifies traditional multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining
- Fine-tunes only the detector head while combining outputs of detector and VLM for each region during inference
- Achieves a remarkable +6.5 mask Average Precision (AP) improvement over previous state-of-the-art on LVIS open-vocabulary detection benchmark
- Competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection tasks
- Ablation studies conducted on backbone fine-tuning, score fusion design/parameters, feature pyramid capacity, and background weight
- Fine-tuning enhances standard detection for base categories but slightly hampers open-vocabulary detection for novel categories
- Effectiveness of using geometric mean over arithmetic mean in score fusion (+8 APr)
- Offers promising approach to object detection by leveraging frozen VLMs effectively

## Summary

In their paper titled "F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models," Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova introduce F-VLM - a novel object detection method that leverages Frozen Vision and Language Models (VLM). This approach simplifies the traditional multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining. The authors make an intriguing observation that a frozen VLM retains crucial locality-sensitive features required for detection and serves as a robust region classifier. The key innovation of F-VLM lies in fine-tuning only the detector head while combining the outputs of the detector and VLM for each region during inference. This approach demonstrates compelling scaling behavior and achieves a remarkable +6.5 mask Average Precision (AP) improvement over the previous state-of-the-art on novel categories of the LVIS open-vocabulary detection benchmark. Additionally, competitive results are showcased on the COCO open-vocabulary detection benchmark and cross-dataset transfer detection tasks. The authors provide insights into their experiments through ablation studies on backbone fine-tuning, score fusion design/parameters, feature pyramid capacity, and background weight. Notably, they find that while fine-tuning enhances standard detection for base categories, it slightly hampers open-vocabulary detection for novel categories. The study also highlights the effectiveness of using geometric mean over arithmetic mean in score fusion (+8 APr), emphasizing the importance of balancing high-scoring regions in both detection and VLM scores simultaneously. Overall,F-VLM presents a promising approach to object detection by leveraging frozen VLMs effectively. The findings not only contribute to advancements in open-vocabulary object detection but also offer insights into potential strategies for further improving performance in this domain. The authors plan to release their code for broader research community access at https://sites.google.com/view/f-vlm/home.