# Supervised Fine-tuning in turn Improves Visual Foundation Models

**Authors:** Xiaohu Jiang, Yixiao Ge, Yuying Ge, Chun Yuan, Ying Shan  
**arXiv:** [2401.10222](https://arxiv.org/abs/2401.10222)  
**Date:** 2024-01-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.10222v1/

## Key points

- The paper discusses the use of supervised fine-tuning (SFT) to improve visual foundation models.
- Visual foundation models are typically pre-trained using methods like CLIP.
- Incorporating region-level visual learning into CLIP's pre-training has been challenging due to a lack of large-scale region-level datasets.
- The authors propose a two-stage method called ViSFT (Vision SFT) that enhances the vision foundation model through visual joint learning on in-domain tasks and evaluates it on out-of-domain benchmarks.
- The results show significant improvements across various out-of-domain benchmarks, including both vision and vision-linguistic scenarios.
- Fine-grained SFT can greatly enhance the performance of vision foundation models by leveraging in-domain tasks and testing on out-of-domain benchmarks.

## Summary

The paper titled "Supervised Fine-tuning in turn Improves Visual Foundation Models" delves into the use of supervised fine-tuning (SFT) to enhance the generation of vision foundation models. These models have been primarily pre-trained using image-text training methods like CLIP in recent years. However, incorporating region-level visual learning into CLIP's pre-training has faced scalability challenges due to the lack of large-scale region-level datasets. To address this issue, the authors propose a two-stage method called ViSFT (Vision SFT) that draws inspiration from SFT in natural language processing. In ViSFT, the vision foundation model is first enhanced through visual joint learning on in-domain tasks and then evaluated on out-of-domain benchmarks. The results show significant improvements across various out-of-domain benchmarks, including both vision and vision-linguistic scenarios. This research demonstrates that fine-grained SFT can greatly enhance the performance of vision foundation models after their pre-training by leveraging in-domain tasks and testing on out-of-domain benchmarks.