# BiFormer: Vision Transformer with Bi-Level Routing Attention

**Authors:** Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, Rynson Lau  
**arXiv:** [2303.08810](https://arxiv.org/abs/2303.08810)  
**Date:** 2023-03-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.08810v1/

## Key points

- Attention is a powerful tool for capturing long-range dependencies, but it can be computationally expensive.
- Previous attempts to introduce sparsity into attention lack flexibility and content awareness.
- The authors propose dynamic sparse attention via bi-level routing to overcome these limitations.
- Bi-level routing attention filters out irrelevant key-value pairs at a coarse region level and applies fine-grained token-to-token attention in the union of remaining candidate regions.
- BiFormer is a new general vision transformer that attends to a small subset of relevant tokens in a query adaptive manner without distraction from other irrelevant ones.
- BiFormer achieves high computational efficiency and good performance in dense prediction tasks such as image classification, object detection, and semantic segmentation.
- Empirical results demonstrate the effectiveness of the proposed design compared to existing methods.
- The authors provide code for their implementation on GitHub.

## Summary

This paper proposes a novel approach to address the computational burden and memory footprint associated with attention in vision transformers. Attention is a powerful tool for capturing long-range dependencies, but it requires pairwise token interaction across all spatial locations, which can be computationally expensive. Previous works have attempted to introduce sparsity into attention through handcrafted and content-agnostic methods such as restricting attention to local windows or axial stripes. However, these approaches lack flexibility and content awareness. To overcome these limitations, the authors propose dynamic sparse attention via bi-level routing. This approach filters out irrelevant key-value pairs at a coarse region level and applies fine-grained token-to-token attention in the union of remaining candidate regions (i.e., routed regions). The proposed bi-level routing attention utilizes sparsity to save both computation and memory while involving only GPU-friendly dense matrix multiplications. Based on this approach, the authors present BiFormer, a new general vision transformer that attends to a small subset of relevant tokens in a query adaptive manner without distraction from other irrelevant ones. BiFormer achieves high computational efficiency and good performance in dense prediction tasks such as image classification, object detection, and semantic segmentation. Empirical results demonstrate the effectiveness of the proposed design compared to existing methods. The authors provide code for their implementation on GitHub. Overall, this paper presents an innovative solution for addressing the computational challenges associated with attention in vision transformers while maintaining high performance levels.