# MoBA: Mixture of Block Attention for Long-Context LLMs

**Authors:** Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu  
**arXiv:** [2502.13189](https://arxiv.org/abs/2502.13189)  
**Date:** 2025-02-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.13189v1/

## Key points

- Paper titled "MoBA: Mixture of Block Attention for Long-Context LLMs" introduces a solution to scaling effective context length in large language models (LLMs) without high computational complexity
- Proposes an approach that leverages the principles of mixture of block attention, allowing models to autonomously determine where to attend without predefined biases
- Offers a novel architecture that excels in long-context tasks and can seamlessly transition between full and sparse attention, enhancing efficiency without compromising performance
- Successfully deployed to support Kimi's long-context requests with superior performance compared to existing approaches
- Code available at https://github.com/MoonshotAI/MoBA for further exploration and implementation
- Represents a significant advancement in efficient attention computation for LLMs, revolutionizing the field of artificial intelligence by enabling effective handling of complex reasoning tasks with extended context lengths

## Summary

In their paper titled "MoBA: Mixture of Block Attention for Long-Context LLMs," Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan,
Suting Xu, Xinran Xu, Guokun Lai,Yanru Chen,Huabin Zheng,Junjie Yan,Jianlin Su,Yuxin Wu
Neo Y. Zhang,Zhilin Yang,Xinyu Zhou,Mingxing Zhang,Jiezhong Qiu introduce a groundbreaking solution to the challenge of scaling effective context length in large language models (LLMs) without incurring prohibitive computational complexity. They address the limitations of traditional attention mechanisms by proposing the  approach which leverages the principles of . The authors emphasize the importance of allowing models to autonomously determine where to attend rather than imposing predefined biases. offers a novel architecture that excels in long-context tasks while providing the flexibility to seamlessly transition between full and sparse attention. This capability enhances efficiency without compromising performance and represents a significant advancement in efficient attention computation for LLMs. Notably,has already been successfully deployed to support Kimi's long-context requests and has demonstrated superior performance compared to existing approaches. The authors make their code available at https://github.com/MoonshotAI/MoBA for further exploration and implementation. Overall,this innovative work showcases how  can revolutionize the field of artificial intelligence by enabling large language models to effectively handle complex reasoning tasks with extended context lengths.