# Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

**Authors:** Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang, Yachen Kang, Sibo Gai, Donglin Wang  
**arXiv:** [2306.12755](https://arxiv.org/abs/2306.12755)  
**Date:** 2023-06-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.12755v1/

## Key points

- Offline reinforcement learning (RL) aims to learn a policy using pre-collected and fixed data, eliminating the need for online interactions.
- Challenges in offline RL include out-of-distribution (OOD) state actions and data inefficiency for training.
- Cross-domain offline RL leverages additional source-domain data to improve training efficiency.
- The BOSA method is introduced to address OOD state actions and OOD transition dynamics issues in cross-domain offline RL.
- BOSA significantly improves offline data efficiency, achieving 74.4% of the state-of-the-art performance with only 10% of the target data.
- BOSA can be easily integrated into model-based offline RL and noising data augmentation techniques.
- Cross domain offline RL with BOSA effectively addresses data inefficiency in traditional methods and enhances practicality in real-world scenarios.

## Summary

Offline reinforcement learning (RL) is a technique that aims to learn a policy using pre-collected and fixed data, eliminating the need for time-consuming online interactions. However, offline RL faces challenges related to out-of-distribution (OOD) state actions and data inefficiency for training. While efforts have been made to address OOD state actions, the issue of data inefficiency has received little attention in offline RL. To tackle this problem, this paper proposes cross-domain offline RL, which assumes that offline data includes additional source-domain data from different transition dynamics or environments. The goal is to leverage this additional data to improve the efficiency of offline training. The authors identify a new challenge in utilizing cross-domain offline data - namely OOD transition dynamics - which goes beyond the common issue of OOD state actions. To address these challenges, the paper introduces a method called BOSA (Supported Cross-Domain Offline Reinforcement Learning), which employs two support-constrained objectives. These objectives aim to mitigate both OOD state actions and OOD transition dynamics issues. The proposed method is evaluated through extensive experiments in the cross-domain offline RL setting. The results demonstrate that BOSA significantly improves offline data efficiency; using only 10% of the target data, BOSA achieves 74.4% of the state-of-the-art (SOTA) performance obtained by using 100% of the target data in offline RL. Additionally, the authors show that BOSA can be easily integrated into model-based offline RL and noising data augmentation techniques used for generating source domain data. This integration naturally avoids potential dynamics mismatch between target domain data and newly generated source domain data. In summary, this paper presents a novel approach called cross domain offline RL to address the issue of data inefficiency in traditional offline RL methods. The proposed method BOSA effectively tackles both OOD state actions and OOD transition dynamics challenges and demonstrates significant improvements in offline data efficiency. The ability to seamlessly integrate BOSA into existing techniques further enhances its practicality and applicability in real world scenarios.