# Auto-scaling Vision Transformers without Training

**Authors:** Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou  
**arXiv:** [2202.11921](https://arxiv.org/abs/2202.11921)  
**Date:** 2022-02-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.11921v2/

## Key points

- The paper "Auto-scaling Vision Transformers without Training" by Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, and Denny Zhou addresses challenges in designing and scaling Vision Transformers (ViTs).
- Two main issues identified are the lack of efficient methods for designing and scaling ViTs and the high computational cost associated with training ViTs compared to convolutional networks.
- As-ViT is introduced as an auto-scaling framework for ViTs that automates the process of discovering and scaling up ViT architectures in a principled and efficient manner.
- As-ViT streamlines the process by automatically finding optimal architectures without extensive human intervention through a training-free search process.
- The approach adjusts widths and depths across different layers of the architecture to create a range of architectures with varying numbers of parameters within a single run.
- A progressive tokenization strategy is introduced to accelerate training and reduce costs by handling coarse tokenization during early training stages.
- As-ViT achieves impressive performance in image classification (83.5% top-1 accuracy on ImageNet-1k) and object detection tasks (52.7% mAP on COCO dataset) without manual intervention or scaling adjustments.
- The end-to-end model design and scaling process using As-ViT only requires 12 hours on a single V100 GPU.
- The work provides valuable insights into automated designing and scaling techniques for Vision Transformers without extensive manual crafting or costly training procedures.
- The code for As-ViT is openly available at https://github.com/VITA-Group/AsViT.

## Summary

The paper "Auto-scaling Vision Transformers without Training" by Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, and Denny Zhou addresses the challenges in designing and scaling Vision Transformers (ViTs). The authors identify two main issues: the lack of efficient methods for designing and scaling ViTs, and the high computational cost associated with training ViTs compared to convolutional networks. To overcome these obstacles, they introduce As-ViT, an auto-scaling framework for ViTs that automates the process of discovering and scaling up ViT architectures in a principled and efficient manner. have gained popularity in recent years as a powerful alternative to traditional convolutional networks for image recognition tasks. However, their design and scaling processes are still relatively manual and time-consuming. This is where  comes into play - it streamlines the process by automatically finding optimal architectures without requiring extensive human intervention. The proposed approach begins by creating a "seed" ViT topology through a training-free search process. This rapid search method is based on a thorough analysis of ViT's network complexity, which shows a strong correlation with ground-truth accuracies using Kendall-tau metrics. Building upon this initial topology, automates the  for ViTs by adjusting widths and depths across different layers of the architecture. This results in a range of architectures with varying numbers of parameters within a single run. Furthermore,the authors introduce a progressive tokenization strategy based on the observation that ViTs can handle coarse tokenization during early training stages. This strategy aims to accelerate and reduce the cost of training ViTs. As-ViT demonstrates impressive performance in both image classification (achieving 83.5% top-1 accuracy on ImageNet-1k) and object detection tasks (with 52.7% mAP on COCO dataset) without manual intervention or scaling adjustments to ViT architectures. Notably, the end-to-end model design and scaling process using As-ViT only requires 12 hours on a single V100 GPU. In conclusion, this work provides valuable insights into automated designing and scaling techniques for  without the need for extensive manual crafting or costly training procedures. The code for As-ViT is openly available at https://github.com/VITA-Group/AsViT. This work was accepted at ICLR 2022 and serves as a significant contribution to the field of computer vision research.