# Go-Explore: a New Approach for Hard-Exploration Problems

**Authors:** Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, Jeff Clune  
**arXiv:** [1901.10995](https://arxiv.org/abs/1901.10995)  
**Date:** 2019-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/1901.10995v1/

## Key points

- Reinforcement learning faces obstacles in intelligent exploration, especially in scenarios with scarce or misleading rewards.
- Atari games Montezuma's Revenge and Pitfall are benchmarks for hard-exploration domains due to their difficulty.
- Existing reinforcement learning algorithms using intrinsic motivation have fallen short on both games.
- Go-Explore algorithm leverages key principles like remembering visited states, prioritizing promising states, and solving simulated environments through methods like determinism and imitation learning.
- Go-Explore has significantly improved performance on hard-exploration problems, achieving remarkable results on Montezuma's Revenge and Pitfall.
- With human-provided domain knowledge, Go-Explore surpasses expectations by achieving "superhuman" performance on Montezuma's Revenge and exceeding expert human performance on Pitfall.
- Go-Explore generates high-performing demonstrations automatically and cost-effectively, surpassing traditional imitation learning approaches where humans provide solution demonstrations.
- The success of Go-Explore opens new research avenues for enhancing capabilities and integrating insights into current reinforcement learning algorithms.

## Summary

Reinforcement learning is a challenging field that faces significant obstacles when it comes to intelligent exploration. This is especially true in scenarios where rewards are scarce or misleading. Two Atari games, Montezuma's Revenge and Pitfall, have been used as benchmarks for hard-exploration domains due to their difficulty. Despite the use of intrinsic motivation in current reinforcement learning algorithms to improve performance in these environments, existing approaches have fallen short on both games. To address this limitation, a novel algorithm called Go-Explore has been introduced. This innovative approach leverages several key principles to enhance exploration efficiency: (1) remembering previously visited states, (2) prioritizing the return to promising states without further exploration before initiating new exploratory actions, and (3) solving simulated environments through various methods such as introducing determinism and robustification through imitation learning. The combination of these principles has resulted in a significant improvement in performance on hard-exploration problems. Notably, Go-Explore has achieved remarkable results on Montezuma's Revenge by scoring a mean of over 43k points – nearly quadrupling the previous state-of-the-art performance. When augmented with human-provided domain knowledge, Go-Explore surpasses expectations by achieving a mean score of over 650k points on Montezuma's Revenge and even outperforms the human world record with a maximum performance close to 18 million points – meeting the criteria for "superhuman" performance. On Pitfall – a notoriously challenging game – Go-Explore with domain knowledge becomes the first algorithm to score above zero and exceeds expert human performance with an impressive mean score of almost 60k points. Furthermore, due to its ability to generate high-performing demonstrations automatically and cost-effectively, Go-Explore surpasses traditional imitation learning approaches where humans provide solution demonstrations. The success of Go-Explore not only opens up new avenues for research into enhancing its capabilities and integrating its insights into current reinforcement learning algorithms, but also holds promise for addressing previously unsolvable hard-exploration problems across various domains – particularly those that utilize simulators during training, such as robotics. In conclusion, with its groundbreaking achievements and potential impact on advancing exploration strategies in reinforcement learning, Go-Explore represents a significant breakthrough in tackling complex challenges within this field.