# GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

**Authors:** Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh  
**arXiv:** [2210.17323](https://arxiv.org/abs/2210.17323)  
**Date:** 2022-10-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.17323v1/

## Key points

- GPT models are recognized for their exceptional performance in complex language modeling tasks
- The large size of GPT models requires multiple high-performance GPUs for inference, limiting usability
- Researchers have explored model compression techniques to address this issue
- Existing compression methods are constrained by the scale and complexity of GPT models
- The authors propose a novel one-shot weight quantization method called GPTQ
- GPTQ leverages approximate second-order information for high accuracy and efficiency
- GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing bitwidth to 3 or 4 bits per weight without significant accuracy degradation compared to uncompressed baseline model
- GPTQ outperforms previously proposed one-shot quantization methods by more than doubling the compression gains while preserving accuracy
- Implementation of GPTQ is available on GitHub for further exploration and application
- Efficient execution of massive language models on a single GPU makes them more accessible for various applications
- Comprehensive evaluation of compression techniques should consider secondary measures beyond standard accuracy metrics, such as transferability and bias effects, to better understand implications
- Funding from European Research Council (ERC) and experimental/compute infrastructure support acknowledged

## Summary

Generative Pre-trained Transformer (GPT) models have gained recognition for their exceptional performance in complex language modeling tasks. However, their extensive computational and storage requirements pose significant challenges. The large size of GPT models necessitates the use of multiple high-performance GPUs for inference, limiting their usability. To address this issue, researchers have explored model compression techniques. However, existing compression methods are constrained by the scale and complexity of GPT models. In this paper, the authors propose a novel one-shot weight quantization method called GPTQ. This method leverages approximate second-order information to achieve both high accuracy and efficiency. The authors demonstrate that GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth to 3 or 4 bits per weight without significant accuracy degradation compared to the uncompressed baseline model. In fact, GPTQ outperforms previously proposed one-shot quantization methods by more than doubling the compression gains while preserving accuracy. The implementation of GPTQ is made available on GitHub for further exploration and application. By enabling efficient execution of massive language models on a single GPU, this research contributes to making these models more accessible for various applications. While the technical details of this work do not raise significant ethical concerns directly, it is important to consider secondary measures beyond standard accuracy metrics when evaluating compressed language models such as transferability and bias effects in order to better understand the implications of compression techniques like GPTQ. The authors acknowledge funding from the European Research Council (ERC) under Horizon 2020 program grant agreement No. 805223 ScaleML as well as experimental support from Eldar Kurtic and compute infrastructure support from Swiss National Supercomputing Center (CSCS). In conclusion, this paper introduces an accurate and efficient weight quantization method called GPTQ which significantly compresses large language models like GPT while highlighting the need for comprehensive evaluation of compression techniques in terms of secondary measures and ethical considerations.