# Deep Reinforcement Learning with Double Q-learning

**Authors:** Hado van Hasselt, Arthur Guez, David Silver  
**arXiv:** [1509.06461](https://arxiv.org/abs/1509.06461)  
**Date:** 2015-09-22  
**Summary page:** https://www.summarizepaper.com/en/paper/1509.06461v1/

## Key points

- The paper addresses the issue of overestimation in the Q-learning algorithm and its impact on performance
- The authors investigate the prevalence of overestimations in practice and their effect on performance
- The study focuses on the DQN algorithm, which combines Q-learning with a deep neural network
- Significant overestimations are observed in certain games within the Atari 2600 domain using the DQN algorithm
- An adaptation of Double Q-learning is proposed to reduce overestimations in large-scale function approximation settings
- Reducing overestimations leads to significantly improved performance on multiple games
- This research highlights important considerations for future developments in deep reinforcement learning algorithms.

## Summary

The paper titled "Deep Reinforcement Learning with Double Q-learning" addresses the issue of overestimation in the popular Q-learning algorithm and its impact on performance. The authors investigate whether such overestimations are common in practice, whether they harm performance, and if there are ways to prevent them. The study focuses on the DQN (Deep Q-Network) algorithm, which combines Q-learning with a deep neural network. The authors demonstrate that the DQN algorithm suffers from significant overestimations in certain games within the Atari 2600 domain. This finding raises concerns about the reliability of the algorithm and its practical applicability. To address this issue, the authors propose an adaptation of the Double Q-learning algorithm, originally introduced in a tabular setting, to work with large-scale function approximation. They show that this adaptation effectively reduces overestimations observed in the DQN algorithm. Furthermore, the authors provide evidence that reducing these overestimations leads to significantly improved performance on multiple games. This finding suggests that addressing overestimation is crucial for achieving better results in reinforcement learning tasks. Overall, this paper contributes to our understanding of the limitations of existing algorithms like DQN and provides a solution through an adapted version of Double Q-learning. By demonstrating both the prevalence of overestimations and their negative impacts on performance, this research highlights important considerations for future developments in deep reinforcement learning algorithms.