# Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions

**Authors:** Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, Jonathan Tompson  
**arXiv:** [2111.14629](https://arxiv.org/abs/2111.14629)  
**Date:** 2021-11-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.14629v1/

## Key points

- Reinforcement learning (RL) agents struggle to generalize their learned behaviors to new scenarios
- Incorporating additional signals beyond the reward function, such as self-supervised learning (SSL), can improve generalization capabilities in RL agents
- Challenges arise when applying online approaches in the offline RL setting
- The study proposes a new framework called Generalized Similarity Functions (GSF) that uses contrastive learning to train an offline RL agent
- GSF aggregates observations based on the similarity of their expected future behavior using generalized value functions
- GSF recovers existing SSL objectives and enhances zero-shot generalization performance on offline Procgen benchmark
- Poor estimation of similarity between observations hinders online algorithms for generalization in the offline setting
- The study provides insights into improving zero-shot generalization in offline reinforcement learning with GSF
- GSF has potential applications in real-world scenarios like autonomous driving at different times of day or night.

## Summary

Reinforcement learning (RL) agents are commonly used to solve complex sequential decision-making tasks. However, these agents often struggle to generalize their learned behaviors to scenarios that were not encountered during training. Previous online approaches have shown that incorporating additional signals beyond the reward function, such as self-supervised learning (SSL), can improve generalization capabilities in RL agents. However, these approaches face challenges when applied in the offline RL setting, where the agent learns from a static dataset. This study addresses the limitations of online algorithms for generalization in RL when applied in the offline setting. The authors propose a new framework called Generalized Similarity Functions (GSF) that uses contrastive learning to train an offline RL agent. GSF aggregates observations based on the similarity of their expected future behavior, quantified using generalized value functions. The authors demonstrate that GSF is versatile enough to recover existing SSL objectives while also enhancing zero-shot generalization performance on a challenging offline RL benchmark called offline Procgen. They highlight that poor estimation of similarity between observations hinders the performance of online algorithms for generalization in the offline setting. The study provides valuable insights into improving zero-shot generalization in offline reinforcement learning by introducing GSF and showcasing its effectiveness on a complex benchmark. The findings contribute to advancing RL techniques and have potential applications in real-world scenarios such as autonomous driving at different times of day or night.