# Offline Reinforcement Learning from Images with Latent Space Models

**Authors:** Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, Chelsea Finn  
**arXiv:** [2012.11547](https://arxiv.org/abs/2012.11547)  
**Date:** 2020-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2012.11547v1/

## Key points

- Offline Reinforcement Learning (RL) is a method of learning policies from a fixed dataset of interactions with the environment.
- It allows for the extensive use and re-use of historical datasets while addressing safety concerns associated with online exploration.
- Previous work in offline RL has focused on tasks with compact state representations, but for real-world applications like robotics, learning directly from rich observation spaces such as images is crucial.
- The authors propose a novel approach that involves learning a latent-state dynamics model and representing uncertainty in the latent space to overcome challenges in quantifying uncertainty in model predictions when dealing with image observations.
- The proposed algorithm significantly outperforms previous offline model-free RL methods and state-of-the-art online visual model-based RL methods in various challenging image-based locomotion and manipulation tasks.
- The approach also excels at an image-based drawer closing task on a real robot using a pre-existing dataset.
- This research introduces a promising method for offline RL from images by leveraging latent space models, showing superior performance compared to existing methods and holding great potential for real-world applications in robotics.

## Summary

Offline Reinforcement Learning (RL) is a method of learning policies from a fixed dataset of interactions with the environment. This approach allows for the extensive use and re-use of historical datasets while also addressing safety concerns associated with online exploration. It has the potential to expand the real-world applicability of RL. Previous work in offline RL has mainly focused on tasks with compact state representations but for real-world applications like robotics, it is crucial to be able to learn directly from rich observation spaces such as images. In this study, the authors build upon recent advancements in model-based algorithms for offline RL and extend them to high-dimensional visual observation spaces. Model-based offline RL algorithms have achieved impressive results in state-based tasks and have strong theoretical guarantees. However, these algorithms heavily rely on the ability to quantify uncertainty in model predictions which becomes particularly challenging when dealing with image observations. To overcome this challenge, the authors propose a novel approach that involves learning a latent-state dynamics model and representing uncertainty in the latent space. This approach is both practical and corresponds to maximizing a lower bound of the Evidence Lower Bound (ELBO) in an unknown Partially Observable Markov Decision Process (POMDP). The authors conducted experiments on various challenging image-based locomotion and manipulation tasks. The results demonstrate that their algorithm significantly outperforms previous offline model-free RL methods as well as state-of-the-art online visual model-based RL methods. Additionally, their approach excels at an image-based drawer closing task on a real robot using a pre-existing dataset. Overall, this research introduces a promising method for offline RL from images by leveraging latent space models. The proposed algorithm shows superior performance compared to existing methods and holds great potential for real world applications in robotics. More information about the experiments and videos can be found on their website [https://sites.google.com/view/lompo/].