# Chain of Thoughtlessness: An Analysis of CoT in Planning

**Authors:** Kaya Stechly, Karthik Valmeekam, Subbarao Kambhampati  
**arXiv:** [2405.04776](https://arxiv.org/abs/2405.04776)  
**Date:** 2024-05-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.04776v1/

## Key points

- Study focuses on performance of large language models (LLMs) on reasoning problems
- Investigates impact of modifying prompts to include examples with chains of thought (CoT) on LLM performance
- Case study conducted using problems from Blocksworld planning domain
- Evaluation based on generality of examples in prompt and complexity of presented problems
- Performance enhancements from CoT prompts observed only when highly specific to problem classes
- Diminishing performance improvements as query-specified stacks grow beyond example sizes
- Emphasizes need for problem-specific prompts over general algorithmic procedures for optimal results
- Significant tradeoff between potential gains and human effort required to generate accurate reasoning traces with CoT approaches

## Summary

In their paper titled "Chain of Thoughtlessness: An Analysis of CoT in Planning," authors Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati delve into the performance of large language models (LLMs) on reasoning problems. Specifically focusing on the concept of chains of thought (CoT), the study aims to investigate whether modifying prompts to include examples with chains of thought can improve LLM performance by teaching the model algorithmic procedures for problem-solving. The authors conduct a case study using problems from Blocksworld, a classical planning domain. They evaluate the performance of two state-of-the-art LLMs along two key axes: the generality of examples provided in the prompt and the complexity of problems presented with each prompt. Despite working with relatively simple problems, the researchers discover that meaningful performance enhancements from chain of thought prompts are only observed when these prompts are highly specific to their respective problem classes. Interestingly, the study reveals that as the size of query-specified stacks grows beyond what is shown in the examples, any performance improvements derived from chain of thought prompts quickly diminish. This finding challenges previous claims in literature suggesting that CoT's effectiveness lies in teaching LLMs general algorithmic procedures through demonstrations. Instead, it highlights a reliance on meticulously crafting problem-specific prompts for optimal results. The authors emphasize a notable drawback associated with chain of thought approaches: a significant tradeoff between potential performance gains and the substantial human effort required to generate examples with accurate reasoning traces. This sheds light on important considerations for leveraging CoT in planning tasks and underscores the importance of carefully tailored prompts for maximizing its benefits.