# Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios

**Authors:** Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan  
**arXiv:** [2207.05501](https://arxiv.org/abs/2207.05501)  
**Date:** 2022-07-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.05501v4/

## Key points

- Most existing vision Transformers (ViTs) struggle to match the efficiency of convolutional neural networks (CNNs) in realistic industrial scenarios
- Hybrid architectures combining CNNs and Transformers have not yielded satisfactory results
- Next-ViT is a next-generation vision Transformer that outperforms both CNNs and ViTs in terms of latency/accuracy trade-off
- Next-ViT introduces two key components: the Next Convolution Block (NCB) and the Next Transformer Block (NTB)
- NCB captures local information using deployment-friendly mechanisms, while NTB captures global information and combines it with local information for enhanced performance
- Next Hybrid Strategy (NHS) efficiently stacks NCB and NTB to improve performance across various vision tasks
- Extensive experiments demonstrate that Next-ViT surpasses existing CNNs, ViTs, and hybrid architectures in terms of latency/accuracy trade-off on TensorRT and CoreML platforms
- Next-ViT models are designed specifically for realistic industrial scenarios and offer improved efficiency compared to previous methods
- Other hybrid architectures like BoTNet, CvT, CMT, Mobile-ViT, Mobile-Former, and EfficientFormer have been proposed in related work
- Core designs within Next-ViT include NCB and NTB blocks for information interaction and modeling short-term/long-term dependencies in visual data
- Fusion of local and global information is performed in the NTB block to enhance modeling capability
- The proposed Next Hybrid Strategy integrates convolutional layers with Transformer blocks innovatively to overcome limitations posed by existing methods

## Summary

In the field of computer vision, most existing vision Transformers (ViTs) struggle to match the efficiency of convolutional neural networks (CNNs) in realistic industrial scenarios. This has led researchers to explore hybrid architectures that combine CNNs and Transformers, but these approaches have not yielded satisfactory results. To address this challenge, the authors propose a next-generation vision Transformer called Next-ViT that outperforms both CNNs and ViTs in terms of latency/accuracy trade-off. Next-ViT introduces two key components: the Next Convolution Block (NCB) and the Next Transformer Block (NTB). The NCB captures local information using deployment-friendly mechanisms, while the NTB captures global information and combines it with local information for enhanced performance. The authors also develop the Next Hybrid Strategy (NHS), which efficiently stacks NCB and NTB to improve performance across various vision tasks. Extensive experiments demonstrate that Next-ViT surpasses existing CNNs, ViTs, and CNN-Transformer hybrid architectures in terms of latency/accuracy trade-off. On TensorRT, Next-ViT achieves significant improvements over ResNet on COCO detection and ADE20K segmentation tasks while maintaining similar latency. On CoreML, it outperforms EfficientFormer on the same tasks under similar latency conditions. The proposed Next-ViT models are designed specifically for realistic industrial scenarios and offer improved efficiency compared to previous methods. The code and models are publicly available for further exploration. In related work, other hybrid architectures such as BoTNet, CvT, CMT, Mobile-ViT, Mobile-Former, and EfficientFormer have been proposed to combine convolutional layers with Transformers in order to leverage their respective strengths for better performance. The core designs within Next-ViT include NCB and NTB blocks for information interaction as well as modeling short-term and long-term dependencies in visual data. Moreover, fusion of local and global information is performed in the NTB block to further enhance modeling capability. Additionally, the proposed Next Hybrid Strategy integrates convolutional layers with Transformer blocks innovatively so as to overcome limitations posed by existing methods. Overall, the proposed architecture offers a promising solution for efficient deployment in realistic industrial scenarios due to its ability to outperform existing CNNs , ViTs ,and hybrid architectures when considering latency/accuracy trade off across various vision tasks .The code and models are publicly available for further exploration .