# When do you need Chain-of-Thought Prompting for ChatGPT?

**Authors:** Jiuhai Chen, Lichang Chen, Heng Huang, Tianyi Zhou  
**arXiv:** [2304.03262](https://arxiv.org/abs/2304.03262)  
**Date:** 2023-04-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03262v2/

## Key points

- Study explores effectiveness of Chain-of-Thought (CoT) prompting on ChatGPT
- CoT has been shown to improve reasoning tasks on LLMs like GPT-3
- Unclear if CoT is still effective on ChatGPT
- CoT is no longer effective for certain tasks like arithmetic reasoning on ChatGPT
- ChatGPT often achieves best performance on these tasks and can generate CoT without explicit instructions
- Suggests ChatGPT may have already been trained with CoT and implicitly memorized the instruction
- Analysis highlights risk of overfitting or bias towards instructions in IFT training of LLMs
- Indicates possible leakage of pretraining recipe allowing verification of dataset and instruction used in training ChatGPT
- Experiments provide new baseline results for ChatGPT on various reasoning tasks
- Offers insights into LLM's profiling, instruction memorization, and pretraining dataset leakage.

## Summary

The study explores the effectiveness of Chain-of-Thought (CoT) prompting on ChatGPT, a more recent instruction finetuned (IFT) Large Language Model (LLM). CoT prompting has been shown to improve the accuracy of reasoning tasks on LLMs like GPT-3. However, it is unclear whether CoT is still effective on ChatGPT. Surprisingly, CoT is no longer effective for certain tasks such as arithmetic reasoning on ChatGPT, while remaining effective for other reasoning tasks. In fact, ChatGPT often achieves the best performance on these tasks and can generate CoT even without explicit instructions. This suggests that ChatGPT may have already been trained with CoT and has implicitly memorized the instruction. The analysis highlights the potential risk of overfitting or bias towards instructions introduced during IFT which is becoming more common in training LLMs. It also indicates possible leakage of the pretraining recipe allowing verification of whether a dataset and instruction were used in training ChatGPT. The experiments conducted in this study provide new baseline results for ChatGPT on various reasoning tasks and offer novel insights into LLM's profiling, instruction memorization and pretraining dataset leakage.