# Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

**Authors:** Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine  
**arXiv:** [1801.01290](https://arxiv.org/abs/1801.01290)  
**Date:** 2018-01-04  
**Summary page:** https://www.summarizepaper.com/en/paper/1801.01290v1/

## Key points

- Model-free algorithms in deep reinforcement learning (RL) show promise in solving complex decision-making and control tasks.
- Two significant challenges faced by these methods are high sample complexity and unstable convergence properties, requiring careful tuning of hyperparameters.
- Soft actor-critic is an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework.
- The actor's objective in this framework is to maximize expected reward while also maximizing entropy, acting as randomly as possible.
- Soft actor-critic achieves state-of-the-art performance on various continuous control benchmark tasks, surpassing both on-policy and off-policy methods used previously.
- One notable advantage of this approach is its stability across different random seeds, consistently achieving similar levels of performance regardless of initial conditions.
- Soft actor critic offers a promising solution to address the challenges faced by model-free deep RL algorithms, making it suitable for real-world applications in diverse domains.

## Summary

In the field of deep reinforcement learning (RL), model-free algorithms have shown promise in solving complex decision-making and control tasks. However, these methods face two significant challenges: high sample complexity and unstable convergence properties, requiring careful tuning of hyperparameters. These limitations restrict their applicability to real-world domains. To address these issues, this paper introduces soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor's objective is twofold: to maximize expected reward while also maximizing entropy. This means that the actor aims to succeed at the task while acting as randomly as possible. Previous deep RL methods based on this framework have been formulated as Q-learning methods. However, by combining off-policy updates with a stable stochastic actor-critic formulation, the proposed soft actor-critic method achieves state-of-the-art performance on various continuous control benchmark tasks, surpassing both on-policy and off-policy methods used previously. One notable advantage of this approach is its stability across different random seeds; unlike other off-policy algorithms that exhibit varying performance depending on initial conditions, soft actor critic consistently achieves similar levels of performance regardless of random seed variations. Overall, the soft actor critic algorithm offers a promising solution to address the challenges faced by model free deep RL algorithms; its ability to achieve high performance in complex control tasks while maintaining stability makes it suitable for real world applications in diverse domains.