# Better & Faster Large Language Models via Multi-token Prediction

**Authors:** Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve  
**arXiv:** [2404.19737](https://arxiv.org/abs/2404.19737)  
**Date:** 2024-04-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.19737v1/

## Key points

- Authors propose a novel approach to training large language models
- Method involves predicting multiple future tokens simultaneously using n independent output heads on top of a shared model trunk
- Auxiliary training task leads to improved downstream capabilities for code and natural language models without additional training time overhead
- Multi-token prediction demonstrated effectiveness on generative benchmarks like coding, outperforming strong baselines by several percentage points
- 13B parameter models solve more problems on HumanEval and MBPP compared to next-token models
- Multi-token prediction enhances induction heads and algorithmic reasoning capabilities for small algorithmic tasks
- Increased speed at inference with up to 3 times faster performance when trained with 4-token prediction

## Summary

The authors of "Better & Faster Large Language Models via Multi-token Prediction," Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve propose a novel approach to training large language models such as GPT and Llama. Their method involves predicting multiple future tokens simultaneously using n independent output heads on top of a shared model trunk. This auxiliary training task leads to improved downstream capabilities for both code and natural language models without any additional training time overhead. The effectiveness of multi-token prediction is demonstrated on generative benchmarks like coding, where their models consistently outperform strong baselines by several percentage points. Notably, their 13B parameter models solve 12% more problems on HumanEval and 17% more on MBPP compared to comparable next-token models. Additionally, experiments show that multi-token prediction enhances induction heads and algorithmic reasoning capabilities for small algorithmic tasks. Another advantage is the increased speed at inference with up to 3 times faster performance even with large batch sizes when trained with 4-token prediction. Detailed performance metrics are provided for abstractive text summarization tasks and the induction capability of n-token prediction models is showcased in the study.