# Optimistic Active Exploration of Dynamical Systems

**Authors:** Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, Andreas Krause  
**arXiv:** [2306.12371](https://arxiv.org/abs/2306.12371)  
**Date:** 2023-06-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.12371v1/

## Key points

- Reinforcement learning algorithms are designed to optimize policies for specific tasks
- Challenge in exploring unknown dynamical systems to estimate models for multiple downstream tasks in a zero-shot manner
- Researchers developed OPAX algorithm for active exploration leveraging probabilistic models to quantify uncertainty
- OPAX takes an optimistic approach to maximize information gain between unknown dynamics and state observations
- Optimization problem reduced to optimal control problem solvable using standard approaches at each episode
- Analyses conducted on general models and specifically explored Gaussian process dynamics
- Sample complexity bound provided, epistemic uncertainty converges to zero with OPAX
- Experimental comparisons show OPAX performs effectively for zero-shot planning on novel downstream tasks
- Active exploration strategies can efficiently navigate unknown dynamical systems and aid in robust policy optimization across diverse tasks

## Summary

In the field of reinforcement learning, algorithms are typically designed to optimize policies for solving specific tasks. However, a key challenge arises when trying to explore an unknown dynamical system in a way that allows for the estimation of models capable of solving multiple downstream tasks in a zero-shot manner. To address this challenge, a team of researchers including Bhavya Sukhija, Lenart Treven, Cansu Sancaktar, Sebastian Blaes, Stelian Coros, and Andreas Krause have developed an algorithm called OPAX for active exploration. OPAX leverages well-calibrated probabilistic models to quantify the epistemic uncertainty surrounding the unknown dynamics of the system. By taking an optimistic approach with respect to plausible dynamics, OPAX aims to maximize the information gain between the unknown dynamics and state observations. This optimization problem is then reduced to an optimal control problem that can be solved at each episode using standard approaches. The researchers conducted analyses on their algorithm for general models and specifically explored Gaussian process dynamics. They were able to provide a sample complexity bound and demonstrate that the epistemic uncertainty converges to zero in this context. Furthermore, experimental comparisons with other heuristic active exploration methods across various environments showed that OPAX not only holds theoretical validity but also performs effectively for zero-shot planning on novel downstream tasks. Overall, the work presented in this paper sheds light on how active exploration strategies can be utilized to efficiently navigate unknown dynamical systems and pave the way for robust policy optimization across diverse tasks.