# Policy Gradient Algorithms Implicitly Optimize by Continuation

**Authors:** Adrien Bolland, Gilles Louppe, Damien Ernst  
**arXiv:** [2305.06851](https://arxiv.org/abs/2305.06851)  
**Date:** 2023-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.06851v3/

## Key points

- Direct policy optimization in reinforcement learning is commonly addressed using policy-gradient algorithms
- These algorithms optimize policy parameters through stochastic gradient ascent
- The paper provides a new theoretical interpretation and justification for these algorithms within the optimization by continuation framework
- Optimizing affine Gaussian policies and incorporating entropy regularization can be viewed as implicitly optimizing deterministic policies through continuation
- Exploration in policy-gradient algorithms involves computing a continuation of the policy's return
- Variance in policies should be history-dependent functions tailored to avoid local extrema rather than solely focusing on maximizing returns

## Summary

, , , , 

In the realm of reinforcement learning, direct policy optimization is commonly tackled using policy-gradient algorithms. These algorithms work to optimize policy parameters through stochastic gradient ascent. This paper offers a fresh theoretical interpretation and justification for these algorithms. The authors first frame direct policy optimization within the optimization by continuation framework, a method for optimizing nonconvex functions by iteratively optimizing surrogate objective functions known as continuations. Furthermore, the authors demonstrate that optimizing affine Gaussian policies and incorporating entropy regularization can be seen as implicitly optimizing deterministic policies through continuation. Building upon these theoretical insights, they propose that exploration in policy-gradient algorithms involves computing a continuation of the policy's return. They also suggest that variance in policies should be history-dependent functions tailored to avoid local extrema rather than solely focusing on maximizing returns. The study was published in Transactions on Machine Learning Research in October 2023 and benefited from valuable discussions with Csaba Szepesvári regarding mathematical aspects that enhanced the quality of the research. The authors express gratitude to their colleagues Gaspard Lambrechts, Arnaud Delaunoy, Pascal Leroy, and Bardhyl Miftari for their insightful comments on the manuscript. Adrien Bolland acknowledges financial support from a research fellowship of the F.R.S.-FNRS.