# Think before you speak: Training Language Models With Pause Tokens

**Authors:** Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, Vaishnavh Nagarajan  
**arXiv:** [2310.02226](https://arxiv.org/abs/2310.02226)  
**Date:** 2023-10-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.02226v1/

## Key points

- The paper introduces the concept of "pause-training" for language model training and inference.
- A learnable "pause" token is added to the input prefix, allowing the model to manipulate more hidden vectors before outputting the next token.
- The approach is evaluated on decoder-only models with 1B and 130M parameters pre-trained on C4 data.
- Empirical evaluation covers various downstream tasks including reasoning, question-answering, general understanding, and fact recall.
- Inference-time delays lead to significant performance gains in language models across multiple tasks when trained with pauses.
- For example, there is an 18% improvement in EM score for the 1B model trained with pauses on the SQuAD QA task.
- Similar improvements are observed on other tasks such as CommonSenseQA and GSM8k reasoning task.
- This work raises conceptual and practical research questions about this new paradigm and its potential applications.

## Summary

The paper titled "Think before you speak: Training Language Models With Pause Tokens" introduces a novel approach to language model training and inference. The authors propose the idea of allowing the model to manipulate a larger number of hidden vectors before outputting the next token by introducing a learnable "pause" token that is appended to the input prefix. This approach, called "pause-training," is evaluated on decoder-only models with 1B and 130M parameters that were pre-trained on C4 data. The empirical evaluation covers various downstream tasks including reasoning, question-answering, general understanding, and fact recall. Results show that inference-time delays lead to significant performance gains in language models across multiple tasks when both pre-trained and fine-tuned with pauses. For example, on the QA task of SQuAD there is an 18% improvement in EM score for the 1B model trained with pauses. Similar improvements are observed on other tasks such as CommonSenseQA and GSM8k reasoning task. This work raises several conceptual and practical research questions regarding this new paradigm and its potential applications in various domains.