# The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

**Authors:** Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, Zhaopeng Tu, Haitao Mi, Dong Yu  
**arXiv:** [2503.02875](https://arxiv.org/abs/2503.02875)  
**Date:** 2025-03-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2503.02875v1/

## Key points

- Authors introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance reasoning capabilities of large language models (LLMs)
- UPFT leverages Prefix Self-Consistency to achieve significant gains in reasoning efficiency without labeled data or exhaustive sampling
- Training exclusively on initial prefix substrings (as few as 8 tokens) yields performance comparable to supervised methods while reducing training time by 75% and sampling cost by 99%
- Errors tend to occur in later stages of reasoning process; prefix-based training effectively preserves model's structural knowledge
- Minimal unsupervised fine-tuning with UPFT can unlock substantial improvements in LLM reasoning capabilities, offering a scalable and resource-efficient alternative to conventional methods

## Summary

In their paper titled "The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models," authors Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, Zhaopeng Tu,
Haitao Mi and Dong Yu introduce a novel approach called Unsupervised Prefix Fine-Tuning (UPFT) to enhance the reasoning capabilities of large language models (LLMs). Traditionally,
improving LLM reasoning efficiency has required supervised fine-tuning with labeled data or computationally expensive sampling. However,
UPFT leverages the observation of Prefix Self-Consistency - the shared initial reasoning steps across diverse solution trajectories - to achieve significant gains in reasoning efficiency without the need for labeled data or exhaustive sampling. By training exclusively on the initial prefix substrings (as few as 8 tokens), UPFT demonstrates performance comparable to supervised methods like Rejection Sampling Fine-Tuning while reducing training time by 75% and sampling cost by 99%. Experimental results on reasoning benchmarks highlight that errors tend to occur in later stages of the reasoning process and that prefix-based training effectively preserves the model's structural knowledge. This innovative approach showcases how minimal unsupervised fine-tuning can unlock substantial improvements in LLM reasoning capabilities,
offering a scalable and resource-efficient alternative to conventional methods. The findings presented in this study have significant implications for advancing the field of natural language processing
and artificial intelligence research.