# Planning Goals for Exploration

**Authors:** Edward S. Hu, Richard Chang, Oleh Rybkin, Dinesh Jayaraman  
**arXiv:** [2303.13002](https://arxiv.org/abs/2303.13002)  
**Date:** 2023-03-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.13002v1/

## Key points

- The paper proposes a new approach to goal-directed exploration in reinforcement learning.
- The authors focus on the goal-conditioned reinforcement learning (GCRL) paradigm, which involves training an agent to achieve goals specified by a reward function.
- They introduce a new method called "Planning Exploratory Goals" (PEG), which sets goals for each training episode to directly optimize an intrinsic exploration reward.
- PEG leverages planning with world models to identify goal commands that will induce high exploration value trajectories, even if they lead to previously observed or physically implausible states.
- PEG enables more efficient and effective training of generalist GCRL policies relative to baselines and ablations in challenging simulated robotics environments.
- The authors discuss related work in the field of exploration in reinforcement learning, highlighting recent approaches that have observed improved exploration but still suffer from shortcomings such as suboptimal launchpads or lack of direct optimization for exploration.
- In conclusion, the proposed PEG method offers a promising solution to the problem of goal-directed exploration in reinforcement learning by optimizing directly for goal selection that generates trajectories with high exploration value.

## Summary

The paper "Planning Goals for Exploration" proposes a novel approach to goal-directed exploration in the context of reinforcement learning. The authors address the problem of how an agent can quickly learn about an unknown environment and accomplish diverse tasks within it. They focus on the goal-conditioned reinforcement learning (GCRL) paradigm, which involves training an agent to achieve goals specified by a reward function. The authors introduce a new method called "Planning Exploratory Goals" (PEG), which sets goals for each training episode to directly optimize an intrinsic exploration reward. PEG leverages planning with world models to identify goal commands that will induce high exploration value trajectories, even if they lead to previously observed or physically implausible states. By doing so, PEG focuses exploration on the most promising parts of the environment and generates interesting training trajectories valuable for policy improvement. To enable this direct optimization, PEG learns world models and adapts sampling-based planning algorithms to plan goal commands. The authors validate their approach on challenging simulated robotics environments including a multi-legged ant robot in a maze and a robot arm on a cluttered tabletop. In both environments, PEG exploration enables more efficient and effective training of generalist GCRL policies relative to baselines and ablations. For instance, their ant successfully navigates a long maze, while their robot arm builds a stack of three blocks upon command. The authors also discuss related work in the field of exploration in reinforcement learning, highlighting recent approaches that have observed improved exploration in long-horizon tasks by extending training episodes or leveraging intrinsic motivation rewards. However, these methods still suffer from shortcomings such as suboptimal launchpads for the subsequent exploration phase or lack of direct optimization for exploration. In conclusion, the proposed PEG method offers a promising solution to the problem of goal-directed exploration in reinforcement learning by optimizing directly for goal selection that generates trajectories with high exploration value. The authors' experiments demonstrate its effectiveness in challenging simulated robotics environments and suggest its potential for real-world applications.