# High-Dimensional Continuous Control Using Generalized Advantage Estimation

**Authors:** John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel  
**arXiv:** [1506.02438](https://arxiv.org/abs/1506.02438)  
**Date:** 2015-06-08  
**Summary page:** https://www.summarizepaper.com/en/paper/1506.02438v1/

## Key points

- The paper addresses the challenge of developing policy gradient methods for high-dimensional state and action spaces.
- The authors propose a scheme called generalized advantage estimation (GAE) to reduce variance and introduce an acceptable level of bias in policy gradient estimates.
- GAE involves estimating the advantage function using a discounted sum of temporal difference residuals, which can be seen as automated cost shaping.
- GAE is compatible with various policy gradient methods and value function approximators, making it straightforward to implement.
- Trust region algorithms are used in conjunction with GAE to optimize the policy and value function, both represented as neural networks.
- Experimental results on 3D locomotion tasks demonstrate that their approach successfully learns complex gaits for bipedal and quadrupedal simulated robots.
- Controllers for bipeds getting up off the ground are also trained using this approach.
- Unlike previous approaches, this work directly maps from raw kinematics to joint torques using neural network policies instead of hand-crafted low-dimensional representations.
- The proposed GAE scheme contributes to reinforcement learning by providing a methodological advancement in policy gradient methods for high-dimensional continuous control problems.

## Summary

This paper, titled "High-Dimensional Continuous Control Using Generalized Advantage Estimation," addresses the challenge of developing policy gradient methods that can effectively handle problems with high-dimensional state and action spaces. The authors propose a scheme called generalized advantage estimation (GAE) that utilizes value functions to reduce the variance of policy gradient estimates while introducing an acceptable level of bias. The GAE scheme involves estimating the advantage function by using a discounted sum of temporal difference residuals. This approach can be seen as a form of automated cost shaping. It is straightforward to implement and compatible with various policy gradient methods and value function approximators. To optimize the policy and value function, both represented as neural networks, trust region algorithms are employed in conjunction with the GAE scheme. The authors present experimental results on challenging 3D locomotion tasks, demonstrating that their approach successfully learns complex gaits for bipedal and quadrupedal simulated robots. Additionally, they showcase the ability to train controllers for bipeds getting up off the ground. A notable aspect of this work is that it differs from previous approaches which rely on hand-crafted low-dimensional policy representations; instead, the neural network policies developed in this study directly map from raw kinematics to joint torques. Overall, this paper contributes to the field of reinforcement learning by providing a methodological advancement in policy gradient methods for high-dimensional continuous control problems. The proposed GAE scheme effectively reduces variance while maintaining an acceptable level of bias, enabling successful learning in challenging scenarios.