# QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models

**Authors:** Elias Frantar, Dan Alistarh  
**arXiv:** [2310.16795](https://arxiv.org/abs/2310.16795)  
**Date:** 2023-10-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.16795v1/

## Key points

- Authors Elias Frantar and Dan Alistarh introduce the QMoE framework to address high inference costs of large language models (LLMs)
- QMoE utilizes Mixture-of-Experts (MoE) architectures for sparse routing, resulting in faster and more accurate models
- The framework offers a scalable algorithm that compresses trillion-parameter MoEs to less than 1 bit per parameter in a custom format co-designed with GPU decoding kernels
- SwitchTransformer-c2048 model can be compressed to less than 160GB (20x compression, 0.8 bits per parameter) with minor accuracy loss achievable in less than a day on a single GPU
- Enables execution of trillion-parameter models on affordable commodity hardware at less than 5% runtime overhead relative to ideal uncompressed inference
- Source code and compressed models for QMoE are available on GitHub at github.com/IST-DASLab/qmoe

## Summary

In their paper "QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models," authors Elias Frantar and Dan Alistarh address the challenge of high inference costs associated with large language models (LLMs) by introducing a new compression and execution framework called QMoE. The framework utilizes Mixture-of-Experts (MoE) architectures to reduce inference costs through sparse routing, resulting in faster and more accurate models. However, deploying models like the SwitchTransformer-c2048 with 1.6 trillion parameters is costly and challenging due to the massive amount of accelerator memory required for efficient operation. The QMoE framework offers a scalable algorithm that accurately compresses trillion-parameter MoEs to less than 1 bit per parameter in a custom format co-designed with GPU decoding kernels. This breakthrough enables end-to-end compressed inference with minimal runtime overhead compared to uncompressed execution. Specifically, the SwitchTransformer-c2048 model can be compressed to less than 160GB (20x compression, 0.8 bits per parameter) with only minor accuracy loss, achievable in less than a day on a single GPU. This advancement allows for the execution of trillion-parameter models on affordable commodity hardware such as a single server equipped with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs at less than 5% runtime overhead relative to ideal uncompressed inference. The source code and compressed models for QMoE are available on GitHub at github.com/IST-DASLab/qmoe. Furthermore, the authors discuss the significance of generative large language models in various practical language and reasoning tasks but highlight their high inference costs as a barrier to widespread deployment. By leveraging MoE architectures and innovative compression techniques like those introduced in QMoE, researchers can overcome memory limitations and reliability roadblocks associated with trillion-parameter models, paving the way for more efficient and cost-effective implementation in real-world applications.