# Mistral 7B

**Authors:** Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed  
**arXiv:** [2310.06825](https://arxiv.org/abs/2310.06825)  
**Date:** 2023-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.06825v1/

## Key points

- Mistral 7B v0.1 is a 7-billion-parameter language model designed for superior performance and efficiency.
- Mistral 7B outperforms Llama 2 13B and Llama 1 34B in reasoning, mathematics, and code generation benchmarks.
- Grouped-query attention (GQA) and sliding window attention (SWA) are utilized to achieve faster inference and handle sequences of arbitrary length with reduced cost.
- Mistral 7B - Instruct is a specialized variant that excels in following instructions and outperforms Llama 2 13B - Chat model on human and automated benchmarks.
- Mistral 7B - Instruct achieves a mean official MT Bench score of 6.84 ± 0.07 over ten iterations, surpassing the official results of Llama 2's score of 6.65.
- Mistral 7B - Instruct can be used as a content moderator to accurately classify user prompts or generated answers as acceptable or falling into categories such as illegal activities or hateful/harassing content.
- The models are released under the Apache 2.0 license, accompanied by their corresponding code available at [https://mistral.ai/news/announcing-mistral-7b/](https://mistral.ai/news/announcing-mistral-7b/).

## Summary

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, including reasoning, mathematics, and code generation. Additionally, Mistral 7B surpasses Llama 1 34B in these areas as well. This achievement is attributed to the utilization of grouped-query attention (GQA) for faster inference and sliding window attention (SWA) to effectively handle sequences of arbitrary length with reduced inference cost. In addition to its impressive performance, Mistral 7B also offers a specialized variant called Mistral 7B - Instruct. This fine-tuned model excels in following instructions and outperforms the Llama 2 13B - Chat model on both human and automated benchmarks. It achieves a mean official MT Bench score of 6.84 ± 0.07 over ten iterations, surpassing the official results of Llama 2's score of 6.65. Furthermore, Mistral 7B - Instruct can be utilized as a content moderator due to its ability to accurately classify user prompts or generated answers as acceptable or falling into categories such as illegal activities (e.g., terrorism, child abuse, fraud) or hateful or harassing content. The models are released under the Apache 2.0 license and are accompanied by their corresponding code which is available at [https://mistral.ai/news/announcing-mistral-7b/](https://mistral.ai/news/announcing-mistral-7b/).