# Proximal Policy Optimization Algorithms

**Authors:** John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov  
**arXiv:** [1707.06347](https://arxiv.org/abs/1707.06347)  
**Date:** 2017-07-20  
**Summary page:** https://www.summarizepaper.com/en/paper/1707.06347v1/

## Key points

- "Proximal Policy Optimization Algorithms" proposes a new family of policy gradient methods for reinforcement learning.
- Methods involve alternating between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent.
- Introduces a novel objective function that allows for multiple epochs of minibatch updates.
- PPO offers advantages over existing approaches such as TRPO: simpler to implement, more general, and better sample complexity based on empirical results.
- Experiments conducted using PPO on benchmark tasks including simulated robotic locomotion and Atari game playing.
- Results demonstrate that PPO outperforms other online policy gradient methods and strikes a favorable balance between sample complexity, simplicity, and wall-time.
- Findings contribute significantly to advancing the field of reinforcement learning by introducing an efficient approach with superior performance while maintaining simplicity in implementation.
- Implications of this work have potential applications for various decision-making systems requiring intelligent decision-making capabilities.

## Summary

The paper titled "Proximal Policy Optimization Algorithms" proposes a new family of policy gradient methods for reinforcement learning. These methods involve alternating between sampling data through interaction with the environment and optimizing a "surrogate" objective function using stochastic gradient ascent. Unlike standard policy gradient methods that perform one gradient update per data sample, the authors introduce a novel objective function that allows for multiple epochs of minibatch updates. The proposed methods, known as proximal policy optimization (PPO), offer several advantages over existing approaches such as trust region policy optimization (TRPO). PPO is simpler to implement, more general, and has better sample complexity based on empirical results. The authors conduct experiments using PPO on various benchmark tasks including simulated robotic locomotion and Atari game playing. The results demonstrate that PPO outperforms other online policy gradient methods and strikes a favorable balance between sample complexity, simplicity, and wall-time. The findings presented in this paper contribute significantly to advancing the field of reinforcement learning by introducing an efficient approach that achieves superior performance while maintaining simplicity in implementation. Furthermore, the implications of this work have potential applications for various decision-making systems requiring intelligent decision-making capabilities.