# ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

**Authors:** Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, Sanjiv Kumar  
**arXiv:** [2312.10003](https://arxiv.org/abs/2312.10003)  
**Date:** 2023-12-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.10003v1/

## Key points

- Minimize human involvement in long-form question answering by leveraging a few labeled demonstrations
- Explore automated tuning of demonstrations through techniques like DSP to optimize prompts using labeled training examples
- Incorporate self-improvement mechanisms driven by AI feedback, diverging from traditional methods
- Methodology stands out for its process-based approach to self-improvement, drawing inspiration from related works such as STAR, ReST, ReSTEM, and RAFT
- Showcase effectiveness of ReST-like strategy in enhancing multi-step reasoning LLM agents through iterative refinement of synthetic data and utilizing datasets like Eli5 and Eli5-askH/askS
- Enable model distillation into smaller yet equally proficient counterparts leading to significant performance improvements
- Ranking "reward" model prioritizes samples based on perplexity minimization to drive trajectory continuation and fine-tuning mixture construction
- Combine ReAct-style reasoning with external knowledge utilization in training multi-step reasoning LLM agents through an innovative self-improvement framework fueled by AI feedback and synthetic data generation

## Summary

Our work focuses on minimizing human involvement in long-form question answering by leveraging only a few labeled demonstrations. We explore automated tuning of these demonstrations through techniques like DSP to optimize prompts using labeled training examples. Our approach diverges from traditional methods by incorporating self-improvement mechanisms driven by AI feedback. Drawing inspiration from related works such as STAR, ReST, ReSTEM, and RAFT, our methodology stands out for its process-based approach to self-improvement. Through iterative refinement of synthetic data and utilizing datasets like Eli5 and Eli5-askH/askS, we showcase the effectiveness of our ReST-like strategy in enhancing multi-step reasoning LLM agents. This not only leads to significant improvements in performance but also enables model distillation into smaller yet equally proficient counterparts. Additionally, our ranking "reward" model prioritizes samples based on perplexity minimization to drive trajectory continuation and fine-tuning mixture construction. In essence, our work showcases the efficacy of combining ReAct-style reasoning with external knowledge utilization in training multi-step reasoning LLM agents through an innovative self-improvement framework fueled by AI feedback and synthetic data generation.