# Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

**Authors:** Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur  
**arXiv:** [2312.06585](https://arxiv.org/abs/2312.06585)  
**Date:** 2023-12-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.06585v1/

## Key points

- Self-training method called ReST$^{EM}$ proposed for fine-tuning language models (LMs) on tasks with scalar feedback
- Method involves generating samples, filtering with binary feedback, and multiple iterations of fine-tuning
- Tested on advanced math reasoning and APPS coding benchmarks using PaLM-2 models
- Outperforms fine-tuning only on human data, scales well with model size
- Performance evaluated on Big-Bench Hard suite of tasks, no major degradation compared to base model
- Problem-solving capabilities assessed on Hungarian high school finals exam in mathematics, performs well compared to existing models except GPT-4
- Ability to generate multiple correct solutions for each problem is a key strength, provides additional training data compared to human-generated data which typically offers only a single solution per problem
- Reduces dependence on human generated data and improves reasoning capabilities in language models

## Summary

In this paper, the authors propose a self-training method called ReST$^{EM}$ to fine-tune language models (LMs) on tasks where scalar feedback is available, such as math problems. The method involves generating samples from the model, filtering them using binary feedback, and fine-tuning the model on these samples through multiple iterations. The authors test ReST$^{EM}$ on advanced math reasoning and APPS coding benchmarks using PaLM-2 models and compare its performance with fine-tuning only on human data. They find that ReST$^{EM}$ scales well with model size and significantly outperforms fine-tuning only on human data. Additionally, they evaluate the performance of ReST$^{EM}$-fine-tuned models on the Big-Bench Hard suite of tasks and find no major degradation compared to the base model. Furthermore, they assess the problem-solving capabilities of their model by evaluating it on a Hungarian high school finals exam in mathematics and find that it performs well, surpassing all existing models except GPT-4. A key strength of ReST$^{EM}$ is its ability to generate multiple correct solutions for each problem, providing valuable additional training data compared to human-generated data which typically offers only a single solution per problem. Through their study, the authors demonstrate that ReST$^{EM}$ can substantially reduce dependence on human generated data and improve reasoning capabilities in language models.