# Improving Intrinsic Exploration by Creating Stationary Objectives

**Authors:** Roger Creus Castanyer, Joshua Romoff, Glen Berseth  
**arXiv:** [2310.18144](https://arxiv.org/abs/2310.18144)  
**Date:** 2023-10-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.18144v1/

## Key points

- Exploration bonuses in reinforcement learning to guide long-horizon exploration
- Limitations of count-based methods in larger state spaces and continuous environments
- Stationary Objectives For Exploration (SOFE) framework to transform non-stationary rewards into stationary ones
- SOFE improves agents' performance in challenging exploration problems
- Experiments demonstrate SOFE's effectiveness in sparse-reward tasks, pixel-based observations, 3D navigation, and procedurally generated environments
- Introduces a novel framework for improving intrinsic exploration in reinforcement learning
- Promising results in addressing challenges related to count-based methods and optimizing agents' objectives in complex environments

## Summary

This paper discusses the use of exploration bonuses in reinforcement learning to guide long-horizon exploration by defining custom intrinsic objectives. It addresses the limitations of count-based methods, which have been shown to perform well in MDPs with a finite and small set of states but introduce unstable learning dynamics in larger state spaces and continuous environments. The authors propose the Stationary Objectives For Exploration (SOFE) framework to transform non-stationary rewards into stationary ones through an augmented state representation. This approach improves agents' performance in challenging exploration problems by simplifying the optimization of their objective. Experiments demonstrate that SOFE enhances agents' performance in various scenarios, including sparse-reward tasks, pixel-based observations, 3D navigation, and procedurally generated environments. Overall, this paper introduces a novel framework for improving intrinsic exploration in reinforcement learning and shows promising results in addressing challenges related to count-based methods and optimizing agents' objectives in complex environments.