# Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

**Authors:** Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, Dongbin Zhao  
**arXiv:** [2505.10832](https://arxiv.org/abs/2505.10832)  
**Date:** 2025-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.10832v1/

## Key points

- Large Reasoning Models (LRMs) can generate explicit reasoning sequences before arriving at final answers
- Detailed reasoning process can lead to computational overhead and latency, especially for simpler problems
- Introducing adaptive thinking in LRMs to dynamically decide if explicit reasoning is necessary based on problem complexity
- Incorporating a simple ellipsis ("...") into the prompt triggers either a thinking or no-thinking mode in the model
- AutoThink is a multi-stage reinforcement learning (RL) framework that optimizes reasoning policies through stage-wise reward shaping
- AutoThink achieves favorable accuracy-efficiency trade-offs compared to recent prompting and RL-based pruning methods on mainstream mathematical benchmarks
- AutoThink improves relative accuracy by 6.4 percent while reducing token usage by 52 percent on DeepSeek-R1-Distill-Qwen-1.5B
- Potential issues with incomplete behavioral separation between thinking and answering, unfiltered training data utilization, and future research directions for performance enhancements

## Summary

In this work, we delve into the realm of Large Reasoning Models (LRMs) and their ability to generate explicit reasoning sequences before arriving at final answers. However, the detailed reasoning process can lead to computational overhead and latency, especially for simpler problems. To tackle this issue of overthinking, we introduce the concept of adaptive thinking in LRMs. By equipping LRMs with the capability to dynamically decide whether explicit reasoning is necessary based on problem complexity, we aim to enhance efficiency without compromising accuracy. Our approach involves incorporating a simple ellipsis ("...") into the prompt, which stochastically triggers either a thinking or no-thinking mode in the model. This reveals a latent controllability in the reasoning behavior of LRMs. Building upon R1-style distilled models, we propose AutoThink – a multi-stage reinforcement learning (RL) framework that optimizes reasoning policies through stage-wise reward shaping. AutoThink learns to engage in explicit reasoning only when essential, defaulting to succinct responses for simpler tasks. Experimental results on five mainstream mathematical benchmarks demonstrate that AutoThink achieves favorable accuracy-efficiency trade-offs compared to recent prompting and RL-based pruning methods. It can seamlessly integrate into any R1-style model, including both distilled and fine-tuned variants. Notably, AutoThink improves relative accuracy by 6.4 percent while reducing token usage by 52 percent on DeepSeek-R1-Distill-Qwen-1.5B, establishing a scalable and adaptive reasoning paradigm for LRMs. While AutoThink showcases promising adaptive reasoning capabilities . These include potential reward hacking issues where incomplete behavioral separation between thinking and answering may occur , and unfiltered training data utilization from the DeepScaleR dataset without task difficulty filtering. Future research directions could explore budget-aware CoT generation and curriculum-based filtering for further performance enhancements. Moreover, additional analyses on reasoning behaviors, training cost considerations, and a case study are presented in Appendix B due to space constraints. The related works section highlights existing RL-based post-training techniques for LLMs and strategies to mitigate overthinking in LRMs through self-generated short CoT signals and pseudo-thinking cues in prompts. In conclusion, our study introduces an innovative approach towards enhancing efficiency in LRMs through adaptive thinking mechanisms guided by multi-stage RL frameworks. By addressing the challenge of overthinking while maintaining performance standards .