# Mixture of A Million Experts

**Authors:** Xu Owen He  
**arXiv:** [2407.04153](https://arxiv.org/abs/2407.04153)  
**Date:** 2024-07-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.04153v1/

## Key points

- Xu Owen He introduces PEER method to address computational costs and memory activation challenges in standard transformer architectures
- PEER leverages fine-grained decomposition and expert retrieval techniques to utilize over a million tiny experts efficiently
- PEER demonstrates superior performance-compute trade-offs compared to dense FFWs and coarse-grained MoEs
- Empirical analysis shows that PEER outperforms existing transformer models while maintaining computational efficiency
- Fine-grained MoE architecture aligns with the scaling law, enabling further scaling of transformer models without sacrificing performance or efficiency

## Summary

In the paper "Mixture of A Million Experts," Xu Owen He introduces a novel approach to address the computational costs and memory activation challenges associated with feedforward (FFW) layers in standard transformer architectures. The proposed method, called PEER, leverages fine-grained decomposition and expert retrieval techniques to efficiently utilize a vast pool of over a million tiny experts. This addresses the limitations of existing sparse mixture-of-experts (MoE) architectures, which are constrained by optimization and computational constraints. By decomposing an extremely wide dense feedforward layer into numerous small experts, PEER demonstrates superior performance-compute trade-offs compared to dense FFWs and coarse-grained MoEs. Empirical analysis conducted on language modeling tasks showcases that PEER outperforms existing transformer models while maintaining computational efficiency. Furthermore, this fine-grained MoE architecture aligns with the scaling law discovered recently, unlocking the potential for further scaling of transformer models without sacrificing performance or computational efficiency. Special thanks are given to Adam Santoro for sharing analysis scripts and Andy Brock for their efforts in building and maintaining the internal codebase used for training the models. Overall, "Mixture of A Million Experts" presents an innovative approach towards enhancing transformer architectures through efficient utilization of a vast pool of experts.