# MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

**Authors:** Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis  
**arXiv:** [2305.07185](https://arxiv.org/abs/2305.07185)  
**Date:** 2023-05-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.07185v2/

## Key points

- Introduces Megabyte, a multi-scale decoder architecture for modeling sequences over one million bytes
- Addresses limitations of autoregressive transformers in scaling to long sequences such as high-resolution images, podcasts, code, or books
- Segments sequences into patches and utilizes local submodel within patches and global model between patches
- Allows for sub-quadratic self-attention, larger feedforward layers, and improved parallelism during decoding
- Better performance at reduced cost for both training and generation
- Byte-level models using Megabyte can compete with subword models in long context language modeling
- Achieves state-of-the-art density estimation on ImageNet and successfully models audio from raw files
- Establishes viability of tokenization-free autoregressive sequence modeling at scale
- Provides additional context regarding dataset sizes and mean document lengths across different modalities such as PG-19, Stories, Books, arXiv, and Code
- Implementation shares similarities with other models to ensure fair comparisons in compute-controlled experiments
- Training procedure involves using the Metaseq2 codebase with PyTorch framework and fairscale for memory efficiency through fully sharded model and optimizer states
- Mixed precision training is employed to improve training efficiency while preserving accuracy at scale
- Reproduced experiments on downsized ImageNet at 64 pixels to validate implementation of PerceiverAR; achieved a bits per byte (bpb) score of 3.53 compared to the reported 3.54 in the original study

## Summary

The paper introduces Megabyte, a multi-scale decoder architecture that enables the modeling of sequences over one million bytes. It addresses the limitations of autoregressive transformers in scaling to long sequences such as high-resolution images, podcasts, code, or books. Megabyte segments sequences into patches and utilizes a local submodel within patches and a global model between patches. This approach allows for sub-quadratic self-attention, larger feedforward layers, and improved parallelism during decoding. The result is better performance at reduced cost for both training and generation. Extensive experiments demonstrate the effectiveness of Megabyte in various tasks. It shows that byte-level models using Megabyte can compete with subword models in long context language modeling. Additionally, it achieves state-of-the-art density estimation on ImageNet and successfully models audio from raw files. These results establish the viability of tokenization-free autoregressive sequence modeling at scale. The paper also provides additional context regarding dataset sizes and mean document lengths across different modalities such as PG-19, Stories, Books, arXiv, and Code. The implementation of Megabyte shares similarities with other models to ensure fair comparisons in compute-controlled experiments. The training procedure involves using the Metaseq2 codebase with PyTorch framework and fairscale for memory efficiency through fully sharded model and optimizer states. Mixed precision training is employed to improve training efficiency while preserving accuracy at scale. In order to validate their implementation of PerceiverAR, the authors reproduce experiments on downsized ImageNet at 64 pixels by matching hyperparameters carefully; they achieve a bits per byte (bpb) score of 3.53 compared to the reported 3.54 in the original study.