# Mamba: Linear-Time Sequence Modeling with Selective State Spaces

**Authors:** Albert Gu, Tri Dao  
**arXiv:** [2312.00752](https://arxiv.org/abs/2312.00752)  
**Date:** 2023-12-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.00752v2/

## Key points

- Authors Albert Gu and Tri Dao introduce Mamba, a novel approach to sequence modeling addressing computational inefficiency of Transformer-based models on long sequences
- Mamba uses selective state spaces within a simplified neural network architecture to improve content-based reasoning, particularly in modalities like language
- Integration of selective state space models (SSMs) in Mamba allows for selective propagation or forgetting of information based on the current token
- Mamba achieves fast inference with 5 times higher throughput than Transformers and linear scaling in sequence length by incorporating a hardware-aware parallel algorithm
- Extensive experimentation across various modalities shows that Mamba outperforms similarly sized Transformers and matches larger Transformers in pretraining and downstream evaluation tasks
- Ablation studies demonstrate that projecting the selection mechanism Δ onto different dimensions significantly impacts model performance, highlighting the importance of fine-tuning model architecture for optimal results

## Summary

In their paper titled "Mamba: Linear-Time Sequence Modeling with Selective State Spaces," authors Albert Gu and Tri Dao introduce a novel approach to sequence modeling that addresses the computational inefficiency of Transformer-based models on long sequences. The traditional attention mechanism used in Transformers has limitations in performing content-based reasoning, particularly on modalities like language. To overcome this challenge, the authors propose the use of selective state spaces within a simplified neural network architecture called Mamba. The key innovation in Mamba is the integration of selective state space models (SSMs) that allow for the selective propagation or forgetting of information along the sequence length dimension based on the current token. By making SSM parameters functions of the input, Mamba can effectively handle discrete modalities and improve performance on real data with million-length sequences. Despite sacrificing efficient convolutions, Mamba incorporates a hardware-aware parallel algorithm in recurrent mode, enabling fast inference with 5 times higher throughput than Transformers and linear scaling in sequence length. Through extensive experimentation across various modalities such as language, audio, and genomics, Mamba demonstrates state-of-the-art performance. In particular, their Mamba-3B model outperforms similarly sized Transformers and matches larger Transformers in both pretraining and downstream evaluation tasks. The authors also discuss related work on selection mechanisms and provide insights into future directions for research in this area. Additionally, ablation studies show that projecting the selection mechanism Δ onto different dimensions significantly impacts model performance, with even a projection to dimension 1 leading to substantial improvements. Further increasing the projection size results in additional enhancements at the cost of slightly more parameters. These findings highlight the importance of fine-tuning model architecture for optimal performance. Overall, Mamba represents a significant advancement in sequence modeling by offering a more efficient alternative to Transformer architectures while achieving superior results across diverse domains. The proposed approach opens up new possibilities for enhancing content-based reasoning capabilities in deep learning applications.