# Ensemble-MIX: Enhancing Sample Efficiency in Multi-Agent RL Using Ensemble Methods

**Authors:** Tom Danino, Nahum Shimkin  
**arXiv:** [2506.02841](https://arxiv.org/abs/2506.02841)  
**Date:** 2025-06-03  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2506.02841v1/

## Key points

- Algorithms in multi-agent reinforcement learning (MARL) have achieved remarkable success in tackling complex tasks
- Common challenge: high volume of environment interactions needed for convergence
- Difficulty in exploring vast joint action spaces and substantial variance present in MARL environments
- Ensemble-MIX algorithm proposed by Tom Danino and Nahum Shimkin addresses these challenges:
- Combines decomposed centralized critic with decentralized ensemble learning
- Selective exploration method leveraging ensemble kurtosis to guide exploration towards states and actions with high uncertainty
- Utilizes diversity-regularized ensemble of individual critics to optimize exploration strategies
- Employs truncated variation of the TD($\lambda$) algorithm for training centralized critic to improve sample efficiency, reduce variance, enhance convergence speed, and stability in training
- Adapts mixed samples approach for actor training by blending on-policy and off-policy loss functions to strike optimal equilibrium between stability and efficiency
- Demonstrated efficacy through rigorous evaluations on standard MARL benchmarks, including diverse SMAC II maps, showcasing superior performance compared to state-of-the-art baselines

## Summary

In the realm of multi-agent reinforcement learning (MARL), algorithms have achieved remarkable success in tackling complex tasks. However, a common challenge persists: the need for a high volume of environment interactions to achieve convergence. This is compounded by the difficulty in exploring vast joint action spaces and the substantial variance present in MARL environments. To address these challenges, Tom Danino and Nahum Shimkin proposed a groundbreaking algorithm known as Ensemble-MIX. Ensemble-MIX introduces a novel approach that combines a decomposed centralized critic with decentralized ensemble learning. It incorporates several innovative contributions to enhance sample efficiency in MARL settings. At its core lies a selective exploration method that leverages ensemble kurtosis to guide exploration towards states and actions with high uncertainty. By extending the global decomposed critic with a diversity-regularized ensemble of individual critics, Ensemble-MIX effectively utilizes excess kurtosis to optimize exploration strategies. To further improve sample efficiency, Ensemble-MIX employs a truncated variation of the TD($\lambda$) algorithm for training the centralized critic. This approach enables efficient off-policy learning while reducing variance, ultimately enhancing convergence speed and stability in training. On the actor side, the algorithm adapts a mixed samples approach to MARL by blending on-policy and off-policy loss functions for actor training. This balanced strategy strikes an optimal equilibrium between stability and efficiency, outperforming purely off-policy learning methods. The efficacy of Ensemble-MIX is demonstrated through rigorous evaluations on standard MARL benchmarks, including diverse SMAC II maps. The results showcase superior performance compared to state-of-the-art baselines, underscoring the effectiveness of this innovative algorithm in enhancing sample efficiency and achieving impressive outcomes in challenging multi-agent environments.