# DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

**Authors:** Xiao-Yin Liu, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Zeng-Guang Hou  
**arXiv:** [2309.08925](https://arxiv.org/abs/2309.08925)  
**Date:** 2023-09-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.08925v1/

## Key points

- The paper proposes a new algorithm called DOMAIN for model-based reinforcement learning in offline settings.
- Offline RL involves learning from a fixed dataset without interacting with the environment.
- Model-based RL learns an environment model from the dataset and generates additional out-of-distribution model data to address distribution shift.
- Previous algorithms rely on estimating model uncertainty for conservatism, but this can be unreliable and result in poor performance.
- DOMAIN introduces an adaptive sampling distribution of model samples to adjust the penalty for using model data, without relying on estimating model uncertainty.
- The Q value learned by DOMAIN outside a specific region is a lower bound of the true Q value, making it less conservative than previous algorithms while guaranteeing security policy improvement.
- Experimental results show that DOMAIN outperforms prior RL algorithms on the D4RL dataset benchmark and other tasks requiring generalization capabilities.

## Summary

The paper titled "DOMAIN: Mildly Conservative Model-Based Offline Reinforcement Learning" proposes a new algorithm called DOMAIN for model-based reinforcement learning (RL) in offline settings. Offline RL involves learning from a fixed dataset without interacting with the environment. Model-based RL learns an environment model from this dataset and generates additional out-of-distribution model data to address the problem of distribution shift. One challenge in model-based RL is the gap between the learned environment model and the actual environment, which can lead to inaccurate predictions. To address this, conservatism needs to be incorporated into the algorithm to balance accurate offline data and imprecise model data. Previous algorithms rely on estimating model uncertainty for conservatism, but uncertainty estimation can be unreliable and result in poor performance in certain scenarios. Additionally, these methods often ignore differences between the model data, leading to excessive conservatism. In response to these issues, DOMAIN introduces an adaptive sampling distribution of model samples that can adjust the penalty for using model data. Unlike previous approaches, DOMAIN does not rely on estimating model uncertainty. The authors theoretically demonstrate that the Q value learned by DOMAIN outside a specific region is a lower bound of the true Q value. This means that DOMAIN is less conservative than previous model-based offline RL algorithms while still guaranteeing security policy improvement. The paper presents extensive experimental results showing that DOMAIN outperforms prior RL algorithms on the D4RL dataset benchmark as well as other tasks requiring generalization capabilities. The authors of this paper are Xiao-Yin Liu, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang , De -Xing Huang , and Zeng -Guang Hou . The paper is categorized under cs .LG (Computer Science - Machine Learning) and cs .AI (Computer Science - Artificial Intelligence). It is 13 pages long and includes 6 figures .