Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

   Benoit Jacob, Skirmantas Kligys, Bo Chen et al. · 2017 · Machine Learning · Not summarized yet

   Abstract

   The rising popularity of intelligent mobile devices and the daunting computational cost of deep learning-based models call for efficient and accurate on-device inference schemes. We propose a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficiently than floating point inference on commonly available integer-only hardware. We also co-design a training procedure to preserve end-to-end model accuracy post quantization. As a result, the proposed quantization scheme improves the tradeoff between accuracy and on-device latency. The improvements are significant even on MobileNets, a model family known for run-time efficienc…

2. Close Not relevant

   Neural Network Quantization for Efficient Inference: A Survey

   Olivia Weng · 2021 · Machine Learning · AI summary available

   Abstract

   As neural networks have become more powerful, there has been a rising desire to deploy them in the real world; however, the power and accuracy of neural networks is largely due to their depth and complexity, making them difficult to deploy, especially in resource-constrained devices. Neural network quantization has recently arisen to meet this demand of reducing the size and complexity of neural networks by reducing the precision of a network. With smaller and simpler networks, it becomes possible to run neural networks within the constraints of their target hardware. This paper surveys the many neural network quantization techniques that have been developed in the last decade. Based on this…

3. Close Not relevant

   DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables

   Darshan C. Ganji, Saad Ashfaq, Ehsan Saboori et al. · 2023 · Machine Learning · AI summary available

   Abstract

   A lot of recent progress has been made in ultra low-bit quantization, promising significant improvements in latency, memory footprint and energy consumption on edge devices. Quantization methods such as Learned Step Size Quantization can achieve model accuracy that is comparable to full-precision floating-point baselines even with sub-byte quantization. However, it is extremely challenging to deploy these ultra low-bit quantized models on mainstream CPU devices because commodity SIMD (Single Instruction, Multiple Data) hardware typically supports no less than 8-bit precision. To overcome this limitation, we propose DeepGEMM, a lookup table based approach for the execution of ultra low-precis…

4. Close Not relevant

   QuIP: 2-Bit Quantization of Large Language Models With Guarantees

   Jerry Chee, Yaohui Cai, Volodymyr Kuleshov et al. · 2023 · Machine Learning · AI summary available

   Abstract

   This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from incoherent weight and Hessian matrices, i.e., from the weights and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale qu…

5. Close Not relevant

   Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

   Burak Bartan, Mert Pilanci · 2021 · Machine Learning · AI summary available

   Abstract

   Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent literature, semidefinite lifting, and Grothendieck's identity. Surprisingly, we show that certain quant…

6. Close Not relevant

   ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization

   Zechun Liu, Changsheng Zhao, Hanxian Huang et al. · 2025 · Machine Learning · AI summary available

   Abstract

   The optimal bit-width for achieving the best trade-off between quantized model size and accuracy has been a subject of ongoing debate. While some advocate for 4-bit quantization, others propose that 1.58-bit offers superior results. However, the lack of a cohesive framework for different bits has left such conclusions relatively tenuous. We present ParetoQ, the first unified framework that facilitates rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. Our findings reveal a notable learning transition between 2 and 3 bits: For 3-bits and above, the fine-tuned models stay close to their original pre-trained distributions, whereas for learning 2-bit netw…

7. Close Not relevant

   8-bit Optimizers via Block-wise Quantization

   Tim Dettmers, Mike Lewis, Sam Shleifer et al. · 2021 · Machine Learning · AI summary available

   Abstract

   Stateful optimizers maintain gradient statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past gradient values. This state can be used to accelerate optimization compared to plain stochastic gradient descent but uses memory that might otherwise be allocated to model parameters, thereby limiting the maximum size of models trained in practice. In this paper, we develop the first optimizers that use 8-bit statistics while maintaining the performance levels of using 32-bit optimizer states. To overcome the resulting computational, quantization, and stability challenges, we develop block-wise dynamic quantization. Block-wise quantization divides…

8. Close Not relevant

   FP4 All the Way: Fully Quantized Training of LLMs

   Brian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Machine Learning · AI summary available

   Abstract

   We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quan…