# Riemannian Proximal Policy Optimization

**Authors:** Shijun Wang, Baocheng Zhu, Chen Li, Mingzhe Wu, James Zhang, Wei Chu, Yuan Qi  
**arXiv:** [2005.09195](https://arxiv.org/abs/2005.09195)  
**Date:** 2020-05-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2005.09195v1/

## Key points

- Shijun Wang et al. propose a Riemannian proximal optimization algorithm for solving MDP problems with guaranteed convergence
- They use a Gaussian mixture model (GMM) to represent policy functions in MDP and formulate it as a nonconvex optimization problem in the Riemannian space of positive semidefinite matrices
- The authors provide a lower bound on policy improvement using bounds derived from the Wasserstein distance of GMMs for two given policy functions
- Schulman et al. introduced TRPO to address challenges in traditional policy gradient methods, ensuring monotonic improvements by constraining KL divergence between old and new policy distributions
- Building upon TRPO, the authors propose PPO which utilizes first-order optimization and clipped probability ratios for improved data efficiency and reliable performance
- The authors leverage Riemannian geometry to develop an efficient optimization algorithm that guarantees convergence for solving MDP problems represented as Gaussian mixture models

## Summary

In this paper, Shijun Wang et al. propose a general Riemannian proximal optimization algorithm for solving Markov decision process (MDP) problems with guaranteed convergence. The authors utilize a Gaussian mixture model (GMM) to represent policy functions in MDP and formulate it as a nonconvex optimization problem in the Riemannian space of positive semidefinite matrices. They also provide a lower bound on policy improvement by using bounds derived from the Wasserstein distance of GMMs for two given policy functions. Preliminary experiments demonstrate the efficacy of their proposed algorithm. Reinforcement learning involves agents exploring and exploiting their environment to maximize long-term rewards, with applications in robot control and game playing. Mainstream methods for reinforcement learning include value iteration and policy gradient methods, which learn optimal policies directly from past experience or on-the-fly. However, traditional policy gradient methods face challenges such as high variance, sample inefficiency, and difficulty in tuning learning rates. To address these challenges, Schulman et al. introduced the trust region policy optimization algorithm (TRPO), which maximizes a surrogate function with constraints on the KL divergence between old and new policy distributions to ensure monotonic improvements. Building upon TRPO, the authors propose the proximal policy optimization algorithm (PPO), which utilizes first-order optimization and clipped probability ratios between new and old policies for improved data efficiency and reliable performance. In reinforcement learning scenarios where policies are represented as Gaussian mixture models, optimizing over positive semidefinite matrices can be challenging due to nonconvexity. The authors' approach leverages Riemannian geometry to develop an efficient optimization algorithm that guarantees convergence for solving MDP problems within this framework. Their method demonstrates promising results in terms of both computational efficiency and effectiveness in improving policies.