# 8-bit Optimizers via Block-wise Quantization

**Authors:** Tim Dettmers, Mike Lewis, Sam Shleifer, Luke Zettlemoyer  
**arXiv:** [2110.02861](https://arxiv.org/abs/2110.02861)  
**Date:** 2021-10-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.02861v2/

## Key points

- Stateful optimizers (e.g., SGD with momentum or Adam) accelerate optimization compared to plain stochastic gradient descent
- These optimizers use memory that could be allocated to model parameters, limiting the maximum size of trained models
- The paper proposes 8-bit optimizers that maintain performance levels achieved by using 32-bit optimizer states
- Block-wise dynamic quantization is developed to overcome computational, quantization, and stability challenges associated with lower-precision statistics
- Dynamic quantization and a stable embedding layer are employed to ensure stability and performance
- The proposed 8-bit optimizers achieve remarkable results on various tasks without changes to original optimizer hyperparameters
- The authors open-source their 8-bit optimizers as a drop-in replacement that only requires a two-line code change
- This approach reduces memory footprint while maintaining 32-bit performance

## Summary

Stateful optimizers, such as SGD with momentum or Adam, maintain gradient statistics over time to accelerate optimization compared to plain stochastic gradient descent. However, these optimizers use memory that could otherwise be allocated to model parameters, limiting the maximum size of trained models. In this paper titled "8-bit Optimizers via Block-wise Quantization," the authors propose a novel approach to address this issue. The authors introduce the first optimizers that utilize 8-bit statistics while maintaining the performance levels achieved by using 32-bit optimizer states. To overcome computational, quantization, and stability challenges associated with using lower-precision statistics, they develop block-wise dynamic quantization. This technique divides input tensors into smaller blocks that are independently quantized and processed in parallel across cores. As a result, it enables faster optimization and high-precision quantization. To ensure stability and performance, the authors combine block-wise quantization with two additional changes. First, they employ dynamic quantization, which is a form of non-linear optimization that is precise for both large and small magnitude values. Second, they introduce a stable embedding layer to reduce gradient variance caused by the highly non-uniform distribution of input tokens in language models. The proposed 8-bit optimizers demonstrate remarkable results on various tasks including language modeling with 1.5B parameters, GLUE finetuning, ImageNet classification, WMT'14 machine translation, MoCo v2 contrastive ImageNet pretraining+finetuning, and RoBERTa pretraining. Importantly, these results are achieved without any changes to the original optimizer hyperparameters. In addition to their findings,<br><br><br><br><br>The authors open-source their 8-bit optimizers as a drop-in replacement that only requires a two-line code change. This makes it easy for researchers and practitioners to adopt their approach and benefit from reduced memory footprint while maintaining 32-bit performance. Overall, this paper presents an innovative solution for reducing memory requirements in stateful optimizers by utilizing 8-bit statistics. The combination of block-wise quantization, dynamic quantization, and a stable embedding layer enables efficient training of large-scale models across various tasks without sacrificing performance.