Results of the summarizing process for the arXiv paper: 2012.04240v1

This paper focuses on the development of a novel FPGA-centric deep neural network (DNN) quantization framework that enables efficient DNN inference engine on FPGA devices through DNN quantization. The proposed solution applies different quantization schemes for different rows of the weight matrix, which is motivated by the fact that the distribution of weights in different rows is not uniform and there is potential to achieve better utilization of heterogeneous FPGA hardware resources. Unlike existing methods that use the same quantization scheme for all weights, this paper proposes a hardware-friendly quantization scheme named sum-of-power-of-2 (SP2), suitable for Gaussian-like weight distribution. The multiplication arithmetic can be replaced with logic shifter and adder, thereby enabling highly efficient implementations with the FPGA LUT resources. In contrast, fixed-point quantization is suitable for Uniform-like weight distribution and can be implemented efficiently by DSP. To fully explore the FPGA resources, an intra-layer multi-scheme quantization framework with an ensemble of SP2 and fixed-point schemes is proposed. This mixed scheme quantization (MSQ) approach can maintain or even increase accuracy due to better matching with weight distributions. The authors evaluate their FPGA-centric quantization framework across multiple application domains with various DNNs such as convolutional neural networks (CNN) and recurrent neural networks (RNN). With optimal SP2/fixed-point ratios on two FPGA devices, i.e., Zynq XC7Z020 and XC7Z045, they achieve performance improvement of 2.1×−4.1× compared to solely exploiting DSPs for all multiplication operations. This research contributes to addressing the critical step of model compression required to deploy DNN models on edge devices due to their huge model size and computation amount. The proposed MSQ approach offers a hardware-friendly solution that enables efficient implementation of DNN inference on edge computing platforms such as ASICs, FPGAs, and embedded systems while maintaining or even improving accuracy. This work is partly supported by the National Science Foundation CCF-1901378, CCF-1919117, CCF-1919289, CNS 1909172 and DARPA HR00112090055 grants.

• - The paper proposes a novel FPGA-centric DNN quantization framework for efficient DNN inference engine on FPGA devices.
• - Different quantization schemes are applied for different rows of the weight matrix to achieve better utilization of heterogeneous FPGA hardware resources.
• - A hardware-friendly quantization scheme named sum-of-power-of-2 (SP2) is proposed for Gaussian-like weight distribution, while fixed-point quantization is suitable for Uniform-like weight distribution.
• - An intra-layer multi-scheme quantization framework with an ensemble of SP2 and fixed-point schemes is proposed to fully explore the FPGA resources and maintain or even increase accuracy due to better matching with weight distributions.
• - The authors evaluate their framework across multiple application domains with various DNNs such as CNN and RNN, achieving performance improvement of 2.1×−4.1× compared to solely exploiting DSPs for all multiplication operations.
• - This research contributes to addressing the critical step of model compression required to deploy DNN models on edge devices while maintaining or even improving accuracy.
• - The proposed MSQ approach offers a hardware-friendly solution that enables efficient implementation of DNN inference on edge computing platforms such as ASICs, FPGAs, and embedded systems.

This paper talks about a new way to make computers learn things faster and use less power. They use a special kind of computer chip called an FPGA. They found that different ways of organizing the information on the chip can make it work better. They tested this idea with different kinds of learning tasks and it worked really well. This is important because it helps us use computers in smaller devices like phones and watches. Definitions- FPGA: A type of computer chip that can be programmed to do specific tasks. - Quantization: A process of reducing the amount of information needed to represent something. - Inference: Using what has been learned to make predictions or decisions. - DSPs: Digital Signal Processors, specialized chips used for processing signals such as audio or video. - Model compression: Reducing the size and complexity of a machine learning model without losing too much accuracy. - Edge devices: Small computing devices that are closer to where data is collected or used, such as smartphones or sensors.

Exploring FPGA-Centric Deep Neural Network Quantization for Efficient Inference on Edge Devices

The development of deep neural networks (DNNs) has enabled the creation of powerful machine learning models that can be used to solve a variety of tasks. However, deploying these models on edge devices is challenging due to their large model size and computation amount. To address this issue, model compression techniques such as quantization are required. This paper focuses on the development of a novel FPGA-centric deep neural network (DNN) quantization framework that enables efficient DNN inference engine on FPGA devices through DNN quantization.

Background

Quantizing a DNN involves reducing the precision of its weights and activations from 32-bit floating point numbers to 8-bit integers or lower bit widths. This reduces both memory requirements and computational complexity, making it possible to deploy DNNs on edge devices with limited resources such as ASICs, FPGAs, and embedded systems. Existing methods typically use the same quantization scheme for all weights in a given layer; however, this approach does not take into account the fact that weight distributions across different rows in a layer may not be uniform. As such, there is potential for better utilization of heterogeneous hardware resources if different quantization schemes are applied to each row based on its distribution type.

Proposed Solution

To explore this potential, the authors propose a hardware-friendly quantization scheme named sum-of-power-of-2 (SP2), which is suitable for Gaussian-like weight distributions. The multiplication arithmetic can be replaced with logic shifter and adder operations using Field Programmable Gate Arrays (FPGAs), thereby enabling highly efficient implementations with minimal LUT resources compared to existing methods which rely solely on Digital Signal Processors (DSP). Fixed point quantization is also proposed as an alternative solution suitable for Uniform like weight distributions; this can be implemented efficiently by leveraging existing DSP cores available in modern FPGAs. In order to fully explore the capabilities of modern FPGAs while maintaining or even increasing accuracy due to better matching with weight distributions, an intra layer multi scheme quantification framework called mixed scheme quantification (MSQ) is proposed which combines SP2 and fixed point schemes together within one layer depending upon their respective suitability for each row’s distribution type . The authors evaluate their proposed MSQ approach across multiple application domains including convolutional neural networks (CNNs) and recurrent neural networks (RNNs). With optimal SP2/fixed point ratios configured across two different types of Xilinx Zynq series 7 field programmable gate arrays – XC7Z020 & XC7Z045 - they achieve performance improvement ranging from 2x - 4x compared against solely exploiting digital signal processors for all multiplication operations .

Conclusion

This research contributes significantly towards addressing the critical step involved in model compression required before deploying DNN models onto edge computing platforms while maintaining or even improving accuracy levels at times . The proposed MSQ approach offers a hardware friendly solution that enables efficient implementation of DNN inference engines onto edge computing platforms such as ASICs , FPGAs & embedded systems . This work was partly supported by grants from National Science Foundation CCF 1901378 , CCF 1919117 , CCF 1919289 , CNS 1909172 & DARPA HR00112090055 respectively .