# To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

**Authors:** Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, Greg Durrett  
**arXiv:** [2409.12183](https://arxiv.org/abs/2409.12183)  
**Date:** 2024-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2409.12183v1/

## Key points

- Chain-of-Thought (CoT) via prompting is a prominent method for eliciting reasoning capabilities from large language models (LLMs).
- A quantitative meta-analysis of over 100 papers and evaluations of 20 datasets across 14 models showed that CoT offers significant performance benefits, especially in tasks involving math or logic.
- CoT enhances symbolic execution but falls short compared to using a symbolic solver.
- CoT is beneficial for applications requiring long-horizon planning and symbolic reasoning, but there is ongoing debate about its effectiveness.
- Variants like tree-of-thought have been explored to address complex planning problems better.
- Leveraging additional calls to LLMs could enhance CoT further, but careful benchmarking is essential to determine the most effective approach.
- Dataset contamination may lead to potential bias in results due to model memorization of answers.
- The meta-analysis concludes that CoT notably improves tasks related to symbolic reasoning, math, and logical reasoning.

## Summary

In the realm of eliciting reasoning capabilities from large language models (LLMs), Chain-of-Thought (CoT) via prompting has emerged as a prominent method. A quantitative meta-analysis covering over 100 papers and evaluations of 20 datasets across 14 models revealed that CoT offers significant performance benefits, particularly in tasks involving math or logic. Interestingly, on tasks like MMLU, generating answers directly without CoT yielded similar accuracy unless symbolic operations were involved. Further analysis showed that CoT's gains stem from enhancing symbolic execution but fall short compared to using a symbolic solver. While CoT proves beneficial for selective applications, especially those requiring long-horizon planning and symbolic reasoning, there is ongoing debate regarding its effectiveness. Variants like tree-of-thought have been explored to address complex planning problems better. Additionally, research suggests that leveraging additional calls to LLMs could enhance CoT further, although careful benchmarking is essential to determine the most effective approach. One limitation highlighted is dataset contamination where models may have memorized answers leading to potential bias in results. However, the study's robustness is supported by the inclusion of diverse language model scales and recent datasets like MuSR and BiGGen Bench. The conclusion drawn from the meta-analysis showcases notable improvements with CoT in tasks related to symbolic reasoning, math, and logical reasoning. Figure 2 illustrates the results aggregated by paper and category showcasing the impact of CoT on different types of tasks. While significant improvements were observed in categories like symbolic reasoning and math with CoT, other categories showed marginal enhancements or no significant difference compared to direct answering methods. Overall this detailed summary highlights the nuanced benefits of CoT in enhancing reasoning capabilities in LLMs across various tasks while emphasizing the need for continued exploration of advanced variants beyond prompt-based approaches.