# Mixtral of Experts

**Authors:** Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak  
**arXiv:** [2401.04088](https://arxiv.org/abs/2401.04088)  
**Date:** 2024-01-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.04088v1/

## Key points

- Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model
- It outperforms previous models in various benchmarks
- Each layer in Mixtral is composed of 8 feedforward blocks or experts
- A router network selects two experts to process the current state and combine their outputs at every token
- Mixtral utilizes an impressive 47B parameters while only using 13B active parameters during inference
- Mixtral excels in mathematics, code generation, and multilingual benchmarks compared to Llama 2 70B
- There is a fine-tuned version called Mixtral 8x7B - Instruct which surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks for following instructions
- Both the base and instruct models are released under the Apache 2.0 license

## Summary

<Mixtral 8x7B>, <Sparse Mixture of Experts (SMoE) language model>, <feedforward blocks>, <context size of 32k tokens>, <fine-tuned version called Mixtral 8x7B - Instruct>

We present Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model that outperforms previous models in various benchmarks. With the same architecture as Mistral 7B, each layer in Mixtral is composed of 8 feedforward blocks or experts. At every token, a router network selects two experts to process the current state and combine their outputs. This allows for access to an impressive 47B parameters while only utilizing 13B active parameters during inference. Notably, Mixtral excels in mathematics, code generation, and multilingual benchmarks compared to Llama 2 70B. Additionally, we offer a fine-tuned version called Mixtral 8x7B - Instruct which surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks for following instructions. Both the base and instruct models are released under the Apache 2.0 license. For more details about Mixtral of Experts, please refer to our publication [link].