# MambaByte: Token-free Selective State Space Model

**Authors:** Junxiong Wang, Tushaar Gangavarapu, Jing Nathan Yan, Alexander M Rush  
**arXiv:** [2401.13660](https://arxiv.org/abs/2401.13660)  
**Date:** 2024-01-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.13660v1/

## Key points

- MambaByte is a token-free adaptation of the Mamba state space model for language modeling
- It learns directly from raw bytes, eliminating bias caused by subword tokenization
- Training MambaByte on byte sequences offers computational efficiency compared to other byte-level models
- MambaByte outperforms state-of-the-art subword Transformers in terms of performance
- MambaByte has linear scaling in length, enabling fast inference and making it suitable for token-free language modeling
- Byte-level language models easily generalize across orthographic and morphological variants
- MambaByte overcomes efficiency challenges faced by autoregressive Transformers when dealing with long byte sequences
- MambaByte maintains computational efficiency comparable to state-of-the-art subword Transformers

## Summary

, , , , 

The paper presents MambaByte, a token-free adaptation of the Mamba state space model for language modeling. Unlike traditional models that rely on subword tokenization, MambaByte learns directly from raw bytes, eliminating any bias. While this approach results in longer sequences, standard autoregressive Transformers struggle with such settings. To address this issue, the authors experiment with training MambaByte on byte sequences and find that it offers computational efficiency compared to other byte-level models. In fact, it even outperforms state-of-the-art subword Transformers. One of the key advantages of MambaByte is its linear scaling in length, which enables fast inference and makes it a viable option for token-free language modeling. Previous studies have highlighted issues with subword tokenizers, such as their lack of robustness to variations in spelling, capitalization, and morphology. In contrast, byte-level language models can easily generalize across orthographic and morphological variants. However, autoregressive Transformers face efficiency challenges due to the quadratic cost of attention when dealing with long byte sequences. But MambaByte overcomes these challenges and offers computational efficiency comparable to state-of-the-art subword Transformers. In summary,<kgd>MambaByte</kgd> proves to be an effective solution for token-free language modeling by leveraging the benefits of byte-level modeling while maintaining computational efficiency similar to state-of-the-art subword Transformers.