# FP8-LM: Training FP8 Large Language Models

**Authors:** Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, Peng Cheng  
**arXiv:** [2310.18313](https://arxiv.org/abs/2310.18313)  
**Date:** 2023-10-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.18313v1/

## Key points

- The paper explores the use of low-bit data formats for efficient training of large language models (LLMs)
- The authors propose a new FP8 automatic mixed-precision framework for LLM training
- The FP8 framework offers three levels of utilization to streamline mixed-precision and distributed parallel training
- Experiment results show that the FP8 framework achieves a 42% reduction in real memory usage and runs 64% faster compared to the BF16 framework
- It surpasses the speed of Nvidia Transformer Engine by 17%
- The FP8 methodology can be applied to other tasks such as LLM instruction tuning and reinforcement learning with human feedback
- The authors have open sourced their FP8 low precision training framework at {https://github.com/Azure/MSAMP}{aka.ms/MSAMP}
- The paper presents a comprehensive exploration into using low bit data formats for efficient LLM training
- The proposed FP8 framework improves memory usage, training speed, and maintains model accuracy
- Its generic applicability makes it valuable for various tasks, and its open sourcing promotes collaboration and further advancements.

## Summary

In this paper titled "FP8-LM: Training FP8 Large Language Models," the authors explore the use of low-bit data formats for efficient training of large language models (LLMs). They propose a new FP8 automatic mixed-precision framework that allows most variables in LLM training, such as gradients and optimizer states, to employ low-precision data formats without compromising model accuracy or requiring changes to hyperparameters. The FP8 framework offers three levels of FP8 utilization to streamline mixed-precision and distributed parallel training for LLMs. It gradually incorporates 8-bit gradients, optimizer states, and distributed learning in an incremental manner. The experiment results demonstrate the effectiveness of the proposed framework. During the training of the GPT-175B model on the H100 GPU platform, the FP8 mixed-precision training framework achieves a remarkable 42% reduction in real memory usage and runs 64% faster than the widely adopted BF16 framework (Megatron-LM). Additionally, it surpasses the speed of Nvidia Transformer Engine by 17%, leading to significant reductions in training costs for large foundation models. Furthermore, the authors highlight that their FP8 mixed-precision training methodology is generic and can be applied seamlessly to other tasks such as LLM instruction tuning and reinforcement learning with human feedback. This offers potential savings in fine-tuning expenses. To facilitate further research and adoption, the authors have open sourced their FP8 low precision training framework at {https://github.com/Azure/MSAMP}{aka.ms/MSAMP}. This allows researchers and practitioners to access and utilize this framework for their own experiments and applications. Overall, this paper presents a comprehensive exploration into using low bit data formats for efficient training of large language models. The proposed FP8 automatic mixed precision framework demonstrates significant improvements in memory usage and training speed while maintaining model accuracy. Its generic applicability makes it a valuable tool for various tasks; moreover its open sourcing promotes collaboration and further advancements in this area.