GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   GPTQT: Quantize Large Language Models Twice to Push the Efficiency

   Yi-Pin Guo, Yi-Lin Lang, Qinyuan Ren · 2024 · 2024 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE International Conference on Rob · cited 5 times · Not summarized yet

   Abstract

   Due to their large size, generative Large Language Models (LLMs) require significant computing and storage resources. This paper introduces a new post-training quantization method, GPTQT, to reduce memory usage and enhance processing speed by expressing the weight of LLM in 3bit/2bit. Practice has shown that minimizing the quantization error of weights is ineffective, leading to overfitting. Therefore, GPTQT employs a progressive two-step approach: initially quantizing weights using Linear quantization to a relatively high bit, followed by converting obtained int weight to lower bit binary coding. A re-explore strategy is proposed to optimize initial scaling factor. During inference, these s…

2. Very close Cites this paper Not relevant

   TR-PTQ: High-Accuracy Integer-Only Transformer Post Training Quantization via Taylor Region Reformulation

   Eli S. Levy, Adam Teman, Yoni Pugachov · 2026 · Not summarized yet

   Abstract

   Post-training quantization (PTQ) enables efficient deployment, yet transformer architectures remain challenging to quantize due to nonlinear layers. While existing methods attribute accuracy loss to insufficient numerical precision, often necessitating floating-point fallbacks, we demonstrate that degradation is actually driven by specific structural error sources. We find that learned scale parameters in normalization layers and compounded approximations in GELU are the primary error contributors, whereas SoftMax remains inherently robust to aggressive quantization. To address these bottlenecks, we introduce TR-PTQ, a unified integer-only formulation using shared Taylor Region (TR) exponent…

3. Very close Not relevant

   Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers

   Junhan Kim, Chungman Lee, Eulrang Cho et al. · 2024 · Neural Information Processing Systems · cited 11 times · Not summarized yet

   Abstract

   With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required. As a cost-effective alternative, learning-free PTQ schemes have been proposed. However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers. In this paper, we thus propose a novel PTQ algo…

4. Very close Cited by this paper Used in depth by this paper A method it uses Not relevant

   ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

   Z. Yao, Reza Yazdani Aminabadi, Minjia Zhang et al. · 2022 · Neural Information Processing Systems · cited 840 times · Not summarized yet

   Abstract

   How to efficiently serve ever-larger trained natural language models in practice has become exceptionally challenging even for powerful cloud servers due to their prohibitive memory/computation requirements. In this work, we present an efficient and affordable post-training quantization approach to compress large Transformer-based models, termed as ZeroQuant. ZeroQuant is an end-to-end quantization and inference pipeline with three main components: (1) a fine-grained hardware-friendly quantization scheme for both weight and activations; (2) a novel affordable layer-by-layer knowledge distillation algorithm (LKD) even without the access to the original training data; (3) a highly-optimized qu…

5. Very close Cites this paper Not relevant

   MPQ: A Message-Passing View of Post-Training Quantization

   Yan-Shuo Chen, Chen-Xi Liu, Reza Shirkavand et al. · 2026 · Not summarized yet

   Abstract

   Post-training quantization (PTQ) is a key technique for efficiently deploying large language models, as it compresses weights into low-bit representations without retraining. Existing PTQ solvers, including GPTQ, LDLQ, and block coordinate descent, commit each weight block to a single codeword before its neighbors are considered, leaving near-optimal alternatives unexplored. In this work, we cast row-wise PTQ as posterior inference over discrete codebooks, in which existing solvers correspond to a zero-temperature, hard-decision limit retaining only the top codeword per block. We propose Message-Passing Quantization (MPQ), which preserves soft posterior information across blocks during infer…

6. Very close Not relevant

   SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models

   Han Liu, Haotian Gao, Xiao-Tong Zhang et al. · 2025 · Knowledge Discovery and Data Mining · cited 7 times · Not summarized yet

   Abstract

   Large language models (LLMs) have shown remarkable performance in various domains, but they are constrained by massive computational and storage costs. Quantization, an effective technique for compressing models to fit resource-limited devices while preserving generative quality, encompasses two primary methods: quantization aware training (QAT) and post-training quantization (PTQ). QAT involves additional retraining or fine-tuning, thus inevitably resulting in high training cost and making it unsuitable for LLMs. Consequently, PTQ has become the research hotspot in recent quantization methods. However, existing PTQ methods usually rely on various complex computation procedures and suffer fr…

7. Very close Another approach to the problem Not relevant

   GPTVQ: The Blessing of Dimensionality for LLM Quantization

   Mart van Baalen, Andrey Kuzmin, Markus Nagel et al. · 2024 · Machine Learning · preprint, not peer reviewed · cited 117 times · AI summary available

   GPTVQ proposes Hessian-guided post-training vector quantization for large language models, addressing the same model-compression problem with a different quantization method.

   Abstract

   In this work we show that the size versus accuracy trade-off of neural network quantization can be significantly improved by increasing the quantization dimensionality. We propose the GPTVQ method, a new fast method for post-training vector quantization (VQ) that scales well to Large Language Models (LLMs). Our method interleaves quantization of one or more columns with updates to the remaining unquantized weights, using information from the Hessian of the per-layer output reconstruction MSE. Quantization codebooks are initialized using an efficient data-aware version of the EM algorithm. The codebooks are then updated, and further compressed by using integer quantization and SVD-based compr…

8. Very close Not relevant

   Compression of Generative Pre-trained Language Models via Quantization

   Chaofan Tao, Lu Hou, Wei Zhang et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cited 120 times · Not summarized yet

   Abstract

   The increasing size of generative Pre-trained Language Models (PLMs) have greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the homogeneous word embeddings caused by reduced capacity and the varied distribution of weights. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning

   Elias Frantar, Dan Alistarh · 2022 · Neural Information Processing Systems · cited 460 times · Not summarized yet

   Abstract

   We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well-performing solutions have been proposed independently for both compression approaches. In this paper, we introduce a new compression framework which covers both weight pruning and quantization in a unified setting, is time- and space-efficient, and …

2. Close Used in depth by this paper A method it uses Not relevant

   BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

   Yuhang Li, Ruihao Gong, Xu Tan et al. · 2021 · International Conference on Learning Representations · cited 772 times · Not summarized yet

   Abstract

   We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Aware Training (QAT). In this work, we propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves a good balance between cross-layer dependency and generalization error. To further employ the power of qu…

3. Close Same question Used in depth by this paper Not relevant

   LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

   Tim Dettmers, Mike Lewis, Younes Belkada et al. · 2022 · Machine Learning · cited 1,403 times · AI summary available

   LLM.int8() reduces inference memory for large transformer language models through 8-bit matrix multiplication while retaining performance.

   Abstract

   Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a t…

4. Close Used in depth by this paper A method it uses Not relevant

   OPT: Open Pre-trained Transformer Language Models

   Susan Zhang, Stephen Roller, Naman Goyal et al. · 2022 · Computation and Language · preprint, not peer reviewed · cited 5,150 times · AI summary available

   Abstract

   Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We…

5. Related* Used in depth by this paper A method it uses Not relevant

   nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models

   Gunho Park, Baeseong Park, S. Kwon et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 77 times · Not summarized yet

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution

   A. Malik, M. Mehra, P. Devan · 2026 · Not summarized yet

   Abstract

   Ultra-low-bit language models promise reductions in storage and memory traffic, but a nominal"1.58-bit"label does not specify the deployed representation or its execution cost. We study a scale-up of an aggressive post-training conversion pipeline from Qwen3-4B to Qwen3-8B. The conversion uses KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation in a weight-only A16 configuration. We do not claim these algorithms as new. Our contribution is the end-to-end scale-up characterisation: an external reproduction gate, matched 4B/8B capability analysis, cross-corpus perplexity, effective-bit accounting, lossless lattice-aware packing, and direct packed execution. The 8B…

2. Close Uses this paper in depth Not relevant

   ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models

   Jia-Qi Chen, Xiang-Long Yan, Cheng-Zhu Bao et al. · 2026 · Not summarized yet

   Abstract

   Mixture-of-Experts (MoE) large language models reduce computation through sparse activation but still incur substantial storage costs. Mixed-precision post-training quantization (PTQ) reduces these costs by quantizing different model components at different bit widths, but existing methods often struggle to preserve generation quality under tight bit budgets. We identify two key challenges: (1) nonlinear quantization errors at extremely low precision are underestimated when cross-layer propagation is ignored; (2) direct router fine-tuning may cause routing drift, and calibration gains may not generalize to downstream tasks. To address these issues, we propose ProQuant, a weight-only PTQ fram…

3. Close Uses this paper in depth Not relevant

   REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent

   Qian Zhang, Yao-Ming Li, Zheng Tan et al. · 2026 · Not summarized yet

   Abstract

   Post-training quantization (PTQ) is essential for deploying large language models (LLMs) under strict resource constraints. State-of-the-art PTQ methods quantize each layer with a single closed-form second-order solver: to remain analytically tractable, they heavily approximate the global loss (dropping cross-channel coupling, pooling output rows into groups), and they then freeze the resulting Hessian across the entire layer, with no way to refresh it as the loss landscape shifts column by column--a phenomenon we call information misalignment. We propose REAL-Q (Real-time E2E-loss Aligned LLM Quantization), a novel PTQ paradigm that breaks this compromise: instead of diluting the objective …

4. Close Not relevant

   OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

   Changhun Lee, Jungyu Jin, Taesu Kim et al. · 2024 · Proceedings of the AAAI Conference on Artificial Intelligence · cited 107 times · Not summarized yet

   Abstract

   Large language models (LLMs) with hundreds of billions of parameters require powerful server-grade GPUs for inference, limiting their practical deployment. To address this challenge, we introduce the outlier-aware weight quantization (OWQ) method, which aims to minimize LLM's footprint through low-precision representation. OWQ prioritizes a small subset of structured weights sensitive to quantization, storing them in high-precision, while applying highly tuned quantization to the remaining dense weights. This sensitivity-aware mixed-precision scheme reduces the quantization error notably, and extensive experiments demonstrate that 3.1-bit models using OWQ perform comparably to 4-bit models o…

5. Close Uses this paper in depth Not relevant

   LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers

   Zhi-Xiong Zhao, Zu-Kang Xu, Xing Hu et al. · 2026 · cited 1 time · Not summarized yet

   Abstract

   Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over evolving recurrent states poses new challenges for low-bit deployment. Existing post-training quantization (PTQ) methods, largely designed for feed-forward Transformers, fail to account for cross-loop activation dynamics, heterogeneous loop sensitivity, and recursive error propagation, leading to substantial performance degradation at low precision. To address these challenges, we propose LoopQuant, a loop-aware PTQ framework for accurate and efficient low-bit quantization of LoopLMs. LoopQuant is built on three core components: 1) Cross-Loop Di…