# Chain of Thought Prompting Elicits Reasoning in Large Language Models

**Authors:** Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, Denny Zhou  
**arXiv:** [2201.11903](https://arxiv.org/abs/2201.11903)  
**Date:** 2022-01-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.11903v1/

## Key points

- Language models struggle with certain reasoning tasks such as math word problems, symbolic manipulation, and commonsense reasoning.
- Inducing a chain of thought via prompting can enable sufficiently large language models to perform better on reasoning tasks that otherwise have flat scaling curves.
- Chain of thought prompting leads to dramatically increasing scaling curves for sufficiently large language models in six reasoning tasks where standard prompting has a flat scaling curve.
- Standard prompting only provides a lower bound on the capabilities of large language models in principle.
- This study leverages prompting by guiding the model to produce self-assisting outputs, unlike most techniques that focus on optimizing inputs/prompts for given tasks or improving interpretability using natural language explanations (NLEs).
- Future work could explore how to induce reasoning at smaller model scales and other prompting methods that might expand the range of tasks that language models can solve.
- The dependence on chain of thought prompting and sufficiently large models are both key components and major limitations.
- Although manually augmenting exemplars with chains of thought is minimal in the few-shot setting, annotation costs could be prohibitive for fine-tuning.
- Chain of thought prompting improves the scaling curve but does not necessarily solve all tasks compared with human accuracy.

## Summary

This paper delves into the ability of language models to generate a coherent chain of thought, which mimics the reasoning process that a person might have when responding to a question. While scaling up language model size has improved performance on various NLP tasks, even the largest models currently struggle with certain reasoning tasks such as math word problems, symbolic manipulation, and commonsense reasoning. The experiments conducted in this study show that inducing a chain of thought via prompting can enable sufficiently large language models to perform better on reasoning tasks that otherwise have flat scaling curves. The emergence of chain of thought reasoning as a consequence of model scale has been a prevailing theme in these experiments. For six reasoning tasks where standard prompting has a flat scaling curve, chain of thought prompting leads to dramatically increasing scaling curves for sufficiently large language models. This observation underscores that standard prompting only provides a lower bound on the capabilities of large language models in principle and raises questions about how much more we can expect the reasoning ability to improve with further increases in model scale. This paper falls under general prompting approaches; however, unlike most techniques that focus on optimizing inputs/prompts for given tasks or improving interpretability using natural language explanations (NLEs), this study leverages prompting by guiding the model to produce self-assisting outputs. Future work could explore how to induce reasoning at smaller model scales and other prompting methods that might expand the range of tasks that language models can solve. The dependence on chain of thought prompting and sufficiently large models are both key components and major limitations. Although manually augmenting exemplars with chains of thought is minimal in the few-shot setting, annotation costs could be prohibitive for fine-tuning. Moreover, although chain of thought prompting improves the scaling curve, it does not necessarily solve all tasks compared with human accuracy. The paper provides valuable insights into the limitations and possibilities of large language models in performing reasoning tasks.