# Scaling Law for Quantization-Aware Training

**Authors:** Mengzhao Chen, Chaoyi Zhang, Jing Liu, Yutao Zeng, Zeyue Xue, Zhiheng Liu, Yunshui Li, Jin Ma, Jie Huang, Xun Zhou, Ping Luo  
**arXiv:** [2505.14302](https://arxiv.org/abs/2505.14302)  
**Date:** 2025-05-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.14302v1/

## Key points

- Large language models (LLMs) pose challenges in terms of computational and memory resources for deployment
- Quantization-aware training (QAT) reduces model precision while maintaining performance
- Existing scaling laws for QAT, particularly at 4-bit precision (W4A4), lack a comprehensive understanding
- A unified scaling law for QAT is introduced in the paper, considering quantization error as a function of model size, training data volume, and quantization group size

## Summary

, , , , 

Large language models (LLMs) present challenges in terms of computational and memory resources, making deployment difficult. Quantization-aware training (QAT) has emerged as a solution to these challenges by reducing model precision while maintaining performance. However, the scaling behavior of QAT, particularly at 4-bit precision (W4A4), lacks a comprehensive understanding. Existing scaling laws for QAT often overlook crucial factors such as the number of training tokens and quantization granularity, limiting their practicality. To address this gap, this paper introduces a unified scaling law for QAT that considers quantization error as a function of model size, training data volume, and quantization group size. Through 268 experiments on QAT, the study reveals that quantization error decreases with increasing model size but increases with more training tokens and coarser quantization granularity. The analysis further decomposes W4A4 quantization error into weight and activation components, both following the overall trend but exhibiting different sensitivities. Specifically, weight quantization error escalates more rapidly with an increase in training tokens. The research identifies activation quantization error in the FC2 layer as the primary bottleneck of W4A4 QAT quantization error due to outliers. By implementing mixed-precision quantization to address this bottleneck, the study demonstrates that weight and activation quantization errors can converge to similar levels. Moreover, with additional training data, weight quantization error eventually surpasses activation quantization error, emphasizing the importance of reducing weight quantization error in such scenarios.