# ZeRO-Offload: Democratizing Billion-Scale Model Training

**Authors:** Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He  
**arXiv:** [2101.06840](https://arxiv.org/abs/2101.06840)  
**Date:** 2021-01-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2101.06840v1/

## Key points

- ZeRO-Offload makes large model training accessible to nearly everyone
- It can train models with over 13 billion parameters on a single GPU, which is a 10x increase in size compared to popular frameworks like PyTorch
- ZeRO-Offload enables large model training by offloading data and compute to CPU while minimizing data movement to/from GPU and reducing CPU compute time
- It maximizes memory savings on GPU, allowing ZeRO-Offload to achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for a 10B parameter model compared to just 30TF using PyTorch alone for a 1.4B parameter model
- ZeRO-Offload can scale on multiple GPUs when available, offering near-linear speedup on up to 128 GPUs
- It can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, which is a 4.5x increase in model size compared to using model parallelism alone
- By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training and makes it accessible even for data scientists with access only to a single GPU.
- The exponential growth in DL model size since the advent of attention-based DL models in 2017 has fueled substantial quality gains.
- The team behind ZeRO Offload includes Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang Minjia Zhang Dong Li and Yuxiong He.
- Their work on ZeRO Offload has resulted in an impressive number of 175 citations as of June 2023.

## Summary

The field of large-scale model training has traditionally been limited to a select few due to the need for complex model refactoring and access to expensive GPU clusters. However, ZeRO-Offload is changing the landscape by making large model training accessible to nearly everyone. This technology can train models with over 13 billion parameters on a single GPU, which is a 10x increase in size compared to popular frameworks like PyTorch, without requiring any changes from data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU while minimizing data movement to/from GPU and reducing CPU compute time. This approach maximizes memory savings on GPU, allowing ZeRO-Offload to achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for a 10B parameter model compared to just 30TF using PyTorch alone for a 1.4B parameter model (the largest that can be trained without running out of memory). Additionally, ZeRO-Offload can scale on multiple GPUs when available, offering near-linear speedup on up to 128 GPUs. Furthermore, ZeRO-Offload can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, which is a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training and makes it accessible even for data scientists with access only to a single GPU. The exponential growth in DL model size since the advent of attention-based DL models in 2017 has fueled substantial quality gains. For example, the largest language model in literature had less than 100M parameters in 2017 but grew rapidly with BERT in 2018 and reached tens of billions by models such as GPT-. Therefore, ZeRO-Offload is crucial for large model training as it democratizes access to this technology and enables data scientists to train models with billions of parameters. The team behind ZeRO-Offload includes Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang Minjia Zhang Dong Li and Yuxiong He. Their work on ZeRO Offload has resulted in an impressive number of 175 citations as of June 2023.