# Curiosity-driven Exploration by Self-supervised Prediction

**Authors:** Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell  
**arXiv:** [1705.05363](https://arxiv.org/abs/1705.05363)  
**Date:** 2017-05-15  
**Summary page:** https://www.summarizepaper.com/en/paper/1705.05363v1/

## Key points

- Using curiosity as an intrinsic reward signal for agents in environments with sparse or absent extrinsic rewards
- Defining curiosity as the error in an agent's ability to predict consequences of its actions in a visual feature space learned through a self-supervised inverse dynamics model
- Efficient exploration in high-dimensional continuous state spaces like images while disregarding irrelevant aspects of the environment
- Evaluation in two diverse environments: VizDoom and Super Mario Bros
- Sparse extrinsic reward: Curiosity enables the agent to reach goals with fewer interactions
- Exploration with no extrinsic reward: Curiosity drives more efficient exploration
- Generalization to unseen scenarios: Prior experience accelerates learning in novel environments
- Promising results in enhancing exploration and skill acquisition without relying heavily on external rewards

## Summary

The paper "Curiosity-driven Exploration by Self-supervised Prediction" by Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell delves into the concept of using curiosity as an intrinsic reward signal for agents operating in environments with sparse or absent extrinsic rewards. The authors propose a novel approach where curiosity is defined as the error in an agent's ability to predict the consequences of its actions in a visual feature space learned through a self-supervised inverse dynamics model. This formulation allows for efficient exploration in high-dimensional continuous state spaces like images while disregarding irrelevant aspects of the environment. The study evaluates this approach in two diverse environments: VizDoom and Super Mario Bros. Three key scenarios are investigated: 1) sparse extrinsic reward, where curiosity enables the agent to reach goals with fewer interactions; 2) exploration with no extrinsic reward, where curiosity drives more efficient exploration; and 3) generalization to unseen scenarios, such as new levels of the same game, where prior experience accelerates learning in novel environments. Overall, the proposed method demonstrates promising results in enhancing exploration and skill acquisition in reinforcement learning tasks without relying heavily on external rewards. The authors provide a demo video and code for further exploration and implementation. This research contributes valuable insights into leveraging curiosity-driven mechanisms for autonomous learning agents operating in challenging real-world scenarios.