# Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency

**Authors:** Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, Zhaoran Wang  
**arXiv:** [2309.17382](https://arxiv.org/abs/2309.17382)  
**Date:** 2023-09-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.17382v2/

## Key points

- Large language models (LLMs) have impressive reasoning abilities
- Translating reasoning into actions in the real world is challenging
- The main issue is completing tasks with minimal interactions using internal reasoning
- Proposed framework called "Reason for Future, Act for Now" (RAFA)
- RAFA orchestrates reasoning and acting with provable regret guarantees
- Prompt template for reasoning that learns from a memory buffer and plans future trajectory
- LLM agent takes initial action of planned trajectory, stores feedback, and replans
- Reasoning in LLMs as learning and planning in Bayesian adaptive Markov decision processes (MDPs)
- Learning and planning performed in an "in-context" manner for actor-critic updates
- Theoretical analysis shows regret proportional to √T (number of interactions)
- Empirical validation shows RAFA outperforms existing frameworks on interactive decision-making tasks
- Focus on tasks where LLM agents optimize actions iteratively by collecting feedback
- Three interleaved modules: learning, planning, and acting in a closed loop
- Core contribution is a principled framework that executes selected subset of planned trajectory to collect feedbacks and replans from new state by learning from feedbacks
- Rigorous mapping between LLMs and reinforcement learning (RL) through RAFA

## Summary

Large language models (LLMs) have shown impressive reasoning abilities, but translating that reasoning into actions in the real world is still challenging. The main issue is how to complete a given task within a minimum number of interactions with the external environment using an internal mechanism of reasoning. In this paper, we propose a framework called "Reason for Future, Act for Now" (RAFA) that orchestrates reasoning and acting in a principled way with provable regret guarantees. To achieve this, we design a prompt template for reasoning that learns from a memory buffer and plans a future trajectory over a long horizon ("reason for future"). At each step, the LLM agent takes the initial action of the planned trajectory ("act for now"), stores the collected feedback in the memory buffer, and reinvokes the reasoning routine to replan the future trajectory from the new state. Our approach casts reasoning in LLMs as learning and planning in Bayesian adaptive Markov decision processes (MDPs). We prompt LLMs to form an updated posterior of the unknown environment from the memory buffer (learning) and generate an optimal trajectory for multiple future steps that maximizes a value function (planning). The learning and planning subroutines are performed in an "in-context" manner to emulate actor-critic updates for MDPs. Theoretical analysis shows that our combination of long-term reasoning and short-term acting achieves regret proportional to √T, where T is the number of interactions with the external environment. This regret bound highlights how prior knowledge obtained through pretraining interacts with uncertainty reduction achieved by reasoning and acting. Empirical validation demonstrates that RAFA outperforms existing frameworks on various interactive decision-making tasks. It achieves nearly perfect scores on benchmarks such as ALFWorld, BlocksWorld, Game of 24, and TicTacToe. Our focus is on interactive decision-making tasks where LLM agents optimize actions iteratively by collecting feedback from the external environment. We operate three interleaved modules - learning, planning, and acting - in a closed loop. While existing graph-search or Monte-Carlo tree-search (MCTS) schemes can be used as planning subroutines for generating trajectories, our core contribution is a principled framework that executes a selected subset of the planned trajectory to collect feedbacks ("act for now") and replans an improved trajectory from the new state by learning from feedbacks ("reason for future"). In conclusion, we establish a rigorous mapping between LLMs and reinforcement learning (RL) through RAFA. Our framework achieves provable regret guarantees and outperforms existing frameworks in interactive decision-making tasks.