# Proximal Policy Optimization Algorithms

**Authors:** John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov  
**arXiv:** [1707.06347](https://arxiv.org/abs/1707.06347)  
**Date:** 2017-07-20  
**Summary page:** https://www.summarizepaper.com/en/paper/1707.06347v2/

## Key points

- "Proximal Policy Optimization Algorithms" proposes a new family of policy gradient methods for reinforcement learning.
- The methods involve alternating between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent.
- Unlike standard policy gradient methods, the proposed methods enable multiple epochs of minibatch updates.
- PPO offers advantages over existing approaches such as TRPO, including simplicity, generality, and better sample complexity in empirical evaluations.
- Experiments on benchmark tasks show that PPO outperforms other online policy gradient methods while maintaining a favorable balance between sample complexity, simplicity, and wall-time.
- PPO is considered a promising approach for reinforcement learning tasks in complex environments.
- The paper provides detailed insights into the methodology and experimental setup, making it valuable for researchers and practitioners in the field.

## Summary

The paper "Proximal Policy Optimization Algorithms" proposes a new family of policy gradient methods for reinforcement learning. These methods involve alternating between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent. Unlike standard policy gradient methods that perform one gradient update per data sample, the authors propose a novel objective function that enables multiple epochs of minibatch updates. The proposed methods, called proximal policy optimization (PPO), offer several advantages over existing approaches such as trust region policy optimization (TRPO). PPO is simpler to implement, more general, and exhibits better sample complexity in empirical evaluations. The authors conduct experiments on various benchmark tasks, including simulated robotic locomotion and Atari game playing, to evaluate the performance of PPO. The results demonstrate that PPO outperforms other online policy gradient methods while striking a favorable balance between sample complexity, simplicity, and wall-time. This suggests that PPO is a promising approach for reinforcement learning tasks. The paper provides detailed insights into the methodology and experimental setup, making it a valuable resource for researchers and practitioners in the field of reinforcement learning. It offers an effective solution to optimize policies in complex environments by combining multiple epochs of minibatch updates with stochastic gradient ascent and improved sample complexity compared to existing approaches like TRPO.