DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables

DeepGEMM is a lookup table based approach proposed by a team of researchers to accelerate ultra low-precision inference on CPU architectures. Recent progress in ultra low-bit quantization has shown significant improvements in latency, memory footprint, and energy consumption on edge devices. To address the limitation of commodity SIMD hardware that typically supports no less than 8-bit precision, DeepGEMM precomputes all possible products of weights and activations and stores them in a lookup table for efficient access at inference time. This method enables the execution of ultra low-precision convolutional neural networks on SIMD hardware with improved performance compared to corresponding 8-bit integer kernels in the QNNPACK framework up to 1.74x on x86 platforms for 2-bit implementation. Moreover, quantization methods such as Learned Step Size Quantization can achieve model accuracy comparable to full-precision floating-point baselines even with sub-byte quantization. Therefore, DeepGEMM promises significant improvements in latency, memory footprint, and energy consumption while maintaining model accuracy for ultra low-bit quantized models deployed on mainstream CPU devices.

DeepGEMM: Accelerating Ultra Low-Precision Inference on CPU Architectures

Recent advances in ultra low-bit quantization have demonstrated significant improvements in latency, memory footprint, and energy consumption on edge devices. To address the limitation of commodity SIMD hardware that typically supports no less than 8-bit precision, a team of researchers proposed DeepGEMM – a lookup table based approach to accelerate ultra low-precision inference on CPU architectures.

What is DeepGEMM?

DeepGEMM is an efficient method for executing ultra low-precision convolutional neural networks (CNNs) on SIMD hardware with improved performance compared to corresponding 8-bit integer kernels in the QNNPACK framework up to 1.74x on x86 platforms for 2-bit implementation. It precomputes all possible products of weights and activations and stores them in a lookup table for efficient access at inference time.

Benefits of DeepGEMM

The main benefit of using DeepGEMM over conventional methods is its ability to achieve model accuracy comparable to full-precision floating point baselines even with sub-byte quantization. This means that it can significantly improve latency, memory footprint, and energy consumption while maintaining model accuracy for ultra low bit quantized models deployed on mainstream CPU devices.

Conclusion

In conclusion, DeepGEMM offers an effective solution for accelerating ultra low precision inference on CPU architectures by precomputing all possible products of weights and activations into a lookup table for efficient access at inference time. This enables the execution of CNNs with improved performance compared to conventional methods while still achieving model accuracy comparable to full precision floating point baselines even with sub byte quantization. As such, it promises significant improvements in latency, memory footprint, and energy consumption when deploying ultra low bit quantized models onto mainstream CPUs devices.