# Scaling laws for post-training quantized large language models

**Authors:** Zifei Xu, Alexander Lan, Wanzin Yazar, Tristan Webb, Sayeh Sharify, Xin Wang  
**arXiv:** [2410.12119](https://arxiv.org/abs/2410.12119)  
**Date:** 2024-10-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.12119v1/

## Key points

- Study title: "Scaling laws for post-training quantized large language models"
- Investigates predictability of post-training weight quantization performance for large language models (LLMs)
- Well-trained LLMs have predictable generalization abilities based on model size
- Quality of compressed LLMs is often unpredictable and requires individual validation
- Conducted systematic empirical study on multiple LLM families using popular weight quantization techniques and low-precision tensor data types
- Identified key scaling factors related to the local loss landscape to predict performance of quantized LLMs
- Developed a statistical model based on these factors
- Explored complexities and trade-offs in post-training weight quantization, highlighting challenges in finding optimal quantization formats and model parameter counts within fixed constraints
- Findings shed light on how properties such as pre-trained negative log-likelihood (NLL) loss scale with total parameter counts in transformer layers' weight tensors
- Provided insights into local radial loss landscape mapping and illustrated trade-off between larger models quantized to lower bit formats versus smaller models quantized to higher bit formats
- Accepted at the 4th NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP) in 2024
- Contributes valuable insights into improving predictability and efficiency of post-training weight quantization for large language models

## Summary

In their study titled "Scaling laws for post-training quantized large language models," Xu et al. investigate the predictability of post-training weight quantization performance for large language models (LLMs). While well-trained LLMs have predictable generalization abilities based on model size, the quality of compressed LLMs is often unpredictable and requires individual validation. To address this issue, the authors conduct a systematic empirical study on multiple LLM families using popular weight quantization techniques and various low-precision tensor data types. They identify key scaling factors related to the local loss landscape that can help predict the performance of quantized LLMs and develop a statistical model based on these factors. The study also delves into the complexities and trade-offs involved in post-training weight quantization, highlighting the challenges in finding optimal quantization formats and model parameter counts within fixed constraints. The authors' findings shed light on how properties such as pre-trained negative log-likelihood (NLL) loss scale with total parameter counts in transformer layers' weight tensors. They also provide insights into local radial loss landscape mapping and illustrate the trade-off between larger models quantized to lower bit formats versus smaller models quantized to higher bit formats. This work was accepted at the 4th NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP) in 2024 and contributes valuable insights into improving the predictability and efficiency of post-training weight quantization for large language models.