# Go-Explore: a New Approach for Hard-Exploration Problems

**Authors:** Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, Jeff Clune  
**arXiv:** [1901.10995](https://arxiv.org/abs/1901.10995)  
**Date:** 2019-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/1901.10995v4/

## Key points

- Significant challenge in reinforcement learning: intelligent exploration
- Benchmark games for hard-exploration domains: Montezuma's Revenge and Pitfall
- Introduction of Go-Explore algorithm with three key principles:
- Remembering previously visited states
- Prioritizing returning to promising states without further exploration before exploring from them
- Tackling simulated environments using any available means and reinforcing solutions through imitation learning
- Go-Explore achieves high scores on Montezuma's Revenge (43k points, exceeding previous state-of-the-art)
- With human-provided domain knowledge, Go-Explore surpasses expectations (average score over 650k points, reaching nearly 18 million points)
- Success on Pitfall game: first algorithm to score above zero, mean score nearly 60k points exceeds expert human performance levels
- Ability to generate high-performing demonstrations automatically and cost-effectively
- Potential for integrating insights from Go-Explore into existing RL algorithms to address challenges in intelligent exploration.

## Summary

In the field of reinforcement learning, a significant challenge lies in intelligent exploration. This is especially true in scenarios where rewards are scarce or misleading. Two Atari games, Montezuma's Revenge and Pitfall, serve as benchmarks for hard-exploration domains and exemplify this difficulty. Despite efforts to enhance performance through intrinsic motivation, current RL algorithms struggle on these games. To address this issue, a novel algorithm called Go-Explore has been introduced. It operates on three key principles: remembering previously visited states, prioritizing returning to promising states without further exploration before exploring from them, and tackling simulated environments using any available means (including introducing determinism) and then reinforcing its solutions through imitation learning. The combination of these principles results in a remarkable improvement in performance on hard-exploration problems. On Montezuma's Revenge, Go-Explore achieves an average score of over 43k points – nearly four times higher than the previous state-of-the-art algorithm. By incorporating human-provided domain knowledge, Go-Explore surpasses expectations with an average score exceeding 650k points and reaching a maximum performance of almost 18 million points – even outperforming human world records. Furthermore, on Pitfall – another challenging game – Go-Explore with domain knowledge becomes the first algorithm to score above zero. Its mean score of nearly 60k points exceeds expert human performance levels. Notably, Go-Explore generates high-performing demonstrations automatically and cost-effectively – surpassing the capabilities of imitation learning methods that rely on human-provided solution demonstrations. The success of Go-Explore not only opens up new avenues for research into enhancing its capabilities but also presents opportunities to integrate its insights into existing RL algorithms. This breakthrough may facilitate progress on previously unsolvable hard-exploration problems across various domains – particularly those utilizing simulators during training such as robotics. Overall, Go-Explore represents a significant advancement in addressing the challenges posed by intelligent exploration in reinforcement learning settings.