# Accessing Vision Foundation Models at ImageNet-level Costs

**Authors:** Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, Yun Fu  
**arXiv:** [2407.10366](https://arxiv.org/abs/2407.10366)  
**Date:** 2024-07-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.10366v1/

## Key points

- Authors address challenges posed by vision foundation models requiring significant training resources and inaccessible training data
- Proposed solution called Proteus aims to distill foundation models into smaller equivalents on ImageNet-1K without original training data access
- Proteus introduces three levels of training objectives (token, patch, feature) to enhance knowledge transfer efficacy
- Achieves remarkable performance at ImageNet-level costs, comparable to DINOv2-L/14 across 15 benchmarks
- Outperforms other vision foundation models like CLIP-L/14, OpenCLIP-L/14, and SynCLR-L/14
- Breakthrough in distillation techniques enhances accessibility to training foundation models and opens up new research possibilities

## Summary

In their paper titled "Accessing Vision Foundation Models at ImageNet-level Costs," authors Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, and Yun Fu address the challenges posed by vision foundation models that require significant training resources and often have inaccessible training data. These models, such as CLIP and DINOv2, are known for their generalization ability due to massive training data but hinder the development of derivatives that could advance research in the field. To overcome these challenges, the authors propose a simple and effective solution called Proteus. <br>
Proteus aims to distill foundation models into smaller equivalents on ImageNet-1K without requiring access to the original training data. By removing designs from conventional knowledge distillation settings that lead to dataset bias, Proteus introduces three levels of training objectives - token, patch, and feature - to enhance knowledge transfer efficacy. This approach allows Proteus to be trained at ImageNet-level costs while still achieving remarkable performance. Utilizing DINOv2-g/14 as the teacher model,<br>
Proteus-L/14 demonstrates comparable performance to the Oracle method DINOv2-L/14 across 15 benchmarks. Impressively,<br>
Proteus outperforms other vision foundation models like CLIP-L/14 (400M), OpenCLIP-L/14 (400M/2B), and SynCLR-L/14 (600M). This breakthrough in distillation techniques not only enhances accessibility to training foundation models but also opens up new possibilities for research in the broader community.