# Latent Exploration for Reinforcement Learning

**Authors:** Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Zixiang Huang, Alexander Mathis  
**arXiv:** [2305.20065](https://arxiv.org/abs/2305.20065)  
**Date:** 2023-05-31  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2305.20065v1/

## Key points

- Reinforcement Learning agents learn policies by exploring and interacting with the environment
- Challenges arise in learning policies mapping high-dimensional sensory input to motor output due to curse of dimensionality
- State-of-the-art methods like SAC and PPO use unstructured exploration through perturbing actuation with independent Gaussian noise during training
- Unstructured exploration may not be optimal for overactuated systems where multiple actuators drive behavior
- Introducing time correlation across action perturbations has been proposed as a solution, but correlation across actuators has been overlooked until now
- Lattice method injects temporally-correlated noise into the latent state of the policy network, integrating with on- and off-policy algorithms
- Structured action noise is modeled as a multivariate Gaussian distribution with full covariance matrix by perturbing network activations
- Experimental results show effectiveness of Lattice in improving performance for complex motor control tasks
- Lattice-SAC achieves state-of-the-art results in PyBullet locomotion tasks, outperforming unstructured exploration by 18% in Humanoid environment
- Lattice-PPO achieves higher rewards in musculoskeletal control environments like MyoSuite, discovering more energy-efficient policies with reductions ranging from 20% to 60%
- Study by Alberto Silvio Chiappa et al. showcases how structured action noise can significantly improve performance in challenging motor control scenarios
- Code for implementing Lattice is available at https://github.com/amathislab/lattice for further exploration and application

## Summary

In the field of Reinforcement Learning, agents learn policies by exploring and interacting with their environment. However, learning policies that map high-dimensional sensory input to motor output can be particularly challenging due to the curse of dimensionality. State-of-the-art methods such as SAC (Soft Actor-Critic) and PPO (Proximal Policy Optimization) address this challenge by exploring the environment through perturbing actuation with independent Gaussian noise during training. While this unstructured exploration has been successful in many tasks, it may not be optimal for overactuated systems where multiple actuators drive behavior. In such cases, uncorrelated perturbations risk diminishing each other's effects or modifying behavior in a task-irrelevant manner. Introducing time correlation across action perturbations has been proposed as a solution to this issue. However, incorporating correlation across actuators has largely been overlooked until now. To address this gap, a novel method called LATent TIme-Correlated Exploration (Lattice) is proposed. Lattice injects temporally-correlated noise into the latent state of the policy network and seamlessly integrates with both on- and off-policy algorithms. This structured action noise is modeled as a multivariate Gaussian distribution with a full covariance matrix by perturbing the network's activations. Experimental results demonstrate the effectiveness of Lattice in improving performance for complex motor control tasks. In PyBullet locomotion tasks, Lattice-SAC achieves state-of-the-art results and outperforms unstructured exploration by 18% in the Humanoid environment. In musculoskeletal control environments like MyoSuite, Lattice-PPO achieves higher rewards in most reaching and object manipulation tasks while also discovering more energy-efficient policies with reductions ranging from 20% to 60%. The study conducted by Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Zixiang Huang, and Alexander Mathis showcases how incorporating structured action noise can significantly improve performance in challenging motor control scenarios. The code for implementing Lattice is available at https://github.com/amathislab/lattice for further exploration and application in reinforcement learning research.