# SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

**Authors:** Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, Yi Ma  
**arXiv:** [2501.17161](https://arxiv.org/abs/2501.17161)  
**Date:** 2025-01-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.17161v1/

## Key points

- Study by Tianzhe Chu et al. compares effects of supervised fine-tuning (SFT) and reinforcement learning (RL) on model generalization capabilities
- Focus on text-based rule variants and visual variants to explore impact of post-training techniques on generalization and memorization
- Introduce GeneralPoints card game and V-IRL navigation environment for assessment
- RL excels at generalizing across rule-based textual and visual variants with outcome-based reward system
- SFT tends to memorize training data, struggles with out-of-distribution scenarios
- RL enhances model's visual recognition capabilities, leading to improved generalization in visual domain
- SFT crucial for effective RL training as it stabilizes model's output format for subsequent improvements
- RL acquires generalizable knowledge in complex multi-modal tasks, highlighting adaptability to diverse scenarios

## Summary

In the study "SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training," Tianzhe Chu et al. investigate the effects of supervised fine-tuning (SFT) and reinforcement learning (RL) on model generalization capabilities. The researchers focus on text-based rule variants and visual variants to explore how these post-training techniques impact generalization and memorization. To assess their impact, they introduce GeneralPoints, an arithmetic reasoning card game, and utilize V-IRL, a real-world navigation environment. The results show that RL excels at generalizing across rule-based textual and visual variants when trained with an outcome-based reward system. In contrast, SFT tends to memorize training data and struggles with out-of-distribution scenarios. Further analysis reveals that RL enhances the model's underlying visual recognition capabilities, leading to improved generalization in the visual domain. While RL shows superior performance in generalization, SFT remains crucial for effective RL training as it stabilizes the model's output format for subsequent improvements. This study highlights RL's ability to acquire generalizable knowledge in complex multi-modal tasks and sheds light on how SFT and RL affect foundation models' adaptability to diverse scenarios.