# GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

**Authors:** Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian  
**arXiv:** [2403.03507](https://arxiv.org/abs/2403.03507)  
**Date:** 2024-03-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.03507v1/

## Key points

- Training Large Language Models (LLMs) faces memory challenges due to increasing weight and optimizer state sizes
- Common techniques like low-rank adaptation (LoRA) have been used, but may require full-rank warm start and alter training dynamics
- Gradient Low-Rank Projection (GaLore) is introduced as a novel strategy for more memory-efficient training compared to LoRA
- GaLore reduces optimizer memory usage by up to 65.5% without compromising efficiency or performance levels
- GaLore's 8-bit implementation further decreases optimizer memory by up to 82.5% and total training memory by 63.3%
- The research aims to enhance memory efficiency in LLM training processes to reduce environmental impact
- GaLore enables larger models to be trained on hardware with lower memory requirements, contributing towards minimizing energy consumption and carbon footprint
- The authors hope that GaLore will inspire future investigations into memory-efficient LLM training strategies using low-rank gradient projection

## Summary

Training Large Language Models (LLMs) poses significant memory challenges due to the increasing size of weights and optimizer states. To address these challenges, common techniques such as low-rank adaptation (LoRA) have been used. However, LoRA restricts the parameter search to a low-rank subspace and may require a full-rank warm start, leading to altered training dynamics. In this study, we introduce Gradient Low-Rank Projection (GaLore), a novel training strategy that enables full-parameter learning while being more memory-efficient than traditional approaches like LoRA. GaLore significantly reduces memory usage by up to 65.5% in optimizer states without compromising efficiency or performance levels for pre-training on LLaMA 1B and 7B architectures using the C4 dataset with up to 19.7 billion tokens. Additionally, GaLore's 8-bit implementation further decreases optimizer memory by up to 82.5% and total training memory by 63.3% compared to a BF16 baseline. This research aims to enhance the memory efficiency of LLM training processes in order to reduce their environmental impact. By enabling larger models to be trained on hardware with lower memory requirements, GaLore contributes towards minimizing energy consumption and carbon footprint associated with LLM pre-training and fine-tuning activities. The authors hope that GaLore will inspire future investigations into memory-efficient LLM training strategies from the perspective of low-rank gradient projection, offering valuable tools for the community to train large language models effectively using consumer-grade hardware resources under limited constraints.