# RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning

**Authors:** Marc Rigter, Bruno Lacerda, Nick Hawes  
**arXiv:** [2204.12581](https://arxiv.org/abs/2204.12581)  
**Date:** 2022-04-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2204.12581v1/

## Key points

- Offline reinforcement learning (RL) aims to find near-optimal policies using pre-collected data without further interaction with the environment
- Model-based algorithms have emerged as a promising approach in offline RL
- RAMBO is a novel approach introduced by the authors
- RAMBO formulates the problem as a two-player zero-sum game against an adversarial environment model
- The model is trained to minimize the value function and accurately predict transitions in the dataset
- RAMBO alternates between optimizing the policy and optimizing the model adversarially to solve the two-player game approximately
- The authors provide a theoretical grounding for their problem formulation, resulting in a PAC performance guarantee and a pessimistic value function that lower bounds the value function in the true environment
- RAMBO achieves state-of-the-art performance on widely studied offline RL benchmarks
- RAMBO presents a robust and effective approach to model-based offline RL by incorporating adversarial training techniques
- Theoretical guarantees and impressive experimental results highlight its potential for real-world applications in various domains.

## Summary

In the field of offline reinforcement learning (RL), the goal is to find near-optimal policies using pre-collected data without further interaction with the environment. Model-based algorithms have emerged as a promising approach to this problem by learning a model of the environment from the dataset and performing conservative policy optimization within that model. In this work, the authors introduce a novel approach called Robust Adversarial Model-Based Offline RL (RAMBO). To achieve conservatism, RAMBO formulates the problem as a two-player zero-sum game against an adversarial environment model. The model is trained to minimize the value function while accurately predicting transitions in the dataset. This forces the policy to act conservatively in areas not covered by the dataset. To solve this two-player game approximately, RAMBO alternates between optimizing the policy and optimizing the model adversarially. The authors provide a theoretical grounding for their problem formulation, resulting in a PAC (probably approximately correct) performance guarantee and a pessimistic value function that lower bounds the value function in the true environment. They evaluate RAMBO on widely studied offline RL benchmarks and demonstrate that it achieves state-of-the-art performance. Overall, RAMBO presents a robust and effective approach to model-based offline RL by incorporating adversarial training techniques. The theoretical guarantees and impressive experimental results highlight its potential for real-world applications in various domains.