# Return-based Scaling: Yet Another Normalisation Trick for Deep RL

**Authors:** Tom Schaul, Georg Ostrovski, Iurii Kemaev, Diana Borsa  
**arXiv:** [2105.05347](https://arxiv.org/abs/2105.05347)  
**Date:** 2021-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2105.05347v1/

## Key points

- Scaling issues are a common frustration in reinforcement learning
- Variability in error scales across domains, tasks, and stages of learning can be significant
- Inconsistency in scaling can impede learning speed and stability, leading to interference between tasks
- Researchers have revisited the topic for agents using temporal-difference learning techniques
- Proposed mechanism aims to address scaling issues without manual tuning, clipping, or adaptation
- Method validated for effectiveness and robustness on Atari games, mitigating interference when training shared neural network on multiple targets with varying reward scales or discounting factors
- Strategies employed by practitioners to navigate scaling challenges include reward and gradient clipping, manipulation of discount factors, non-linear reward transformations, use of separate networks instead of shared architecture
- Some methods confront scale issue directly while others manifest as subtle "tricks of the trade"
- Research delves into scenarios where scaling plays pivotal role and outlines criteria for effective normalization schemes

## Summary

In the realm of reinforcement learning, scaling issues are a common frustration for practitioners. The variability in error scales across different domains, tasks, and stages of learning can be significant, sometimes spanning orders of magnitude. This inconsistency can impede the speed and stability of learning processes and lead to interference between tasks. It also necessitates extensive tuning efforts. In response to these challenges, researchers have revisited the topic specifically for agents utilizing temporal-difference learning techniques. One proposed mechanism aims to address scaling issues without the need for manual tuning, clipping or adaptation. This method has been validated for its effectiveness and robustness on a range of Atari games. Particularly noteworthy is its ability to mitigate interference when training a shared neural network on multiple targets with varying reward scales or discounting factors. The exploration of scaling challenges in deep reinforcement learning has uncovered various strategies employed by practitioners to navigate these complexities. These include reward and gradient clipping, manipulation of discount factors, non-linear reward transformations as well as the use of separate networks instead of a shared architecture. While some methods directly confront the issue of scale head-on, others manifest as subtle "tricks of the trade" within the field. This work delves into prototypical scenarios where scaling plays a pivotal role and outlines criteria for effective normalization schemes. By highlighting instances where simple fixes fall short in addressing scaling issues effectively, this research contributes valuable insights to the ongoing discourse surrounding optimal strategies for managing scale-related challenges in deep reinforcement learning applications.