# Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

**Authors:** Yarin Gal, Zoubin Ghahramani  
**arXiv:** [1506.02142](https://arxiv.org/abs/1506.02142)  
**Date:** 2015-06-06  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1506.02142v1/

## Key points

- Deep learning tools are effective in regression and classification tasks but often lack the ability to capture model uncertainty.
- Bayesian models provide a solution for reasoning about model uncertainty, but they can be computationally expensive.
- Dropout in multilayer perceptron models can serve as a Bayesian approximation, allowing for modeling uncertainty without compromising computational performance or test accuracy.
- Incorporating model uncertainty is particularly important for classification tasks, as shown with the MNIST dataset.
- The study explores the properties of dropout uncertainty across different network architectures and non-linearities, showcasing its impact on tasks like extrapolation, interpolation, and classification.
- Model uncertainty can be integrated into a Bayesian pipeline with deep reinforcement learning as an application example, enhancing predictive accuracy and decision-making processes in machine learning applications.

## Summary

In the realm of applied machine learning, deep learning tools have garnered significant attention for their effectiveness in regression and classification tasks. However, a common limitation of these tools is the inability to capture model uncertainty. This uncertainty is crucial for making informed decisions based on predictions. To address this challenge, Bayesian models offer a solution by allowing us to reason about model uncertainty. However, they often come with a high computational cost that can be prohibitive. In a groundbreaking study by Yarin Gal and Zoubin Ghahramani, it was demonstrated that dropout in multilayer perceptron models (MLPs) can serve as a Bayesian approximation. By leveraging dropout as a means to model uncertainty in MLPs, the researchers were able to extract valuable information that had previously been discarded from existing models. This innovative approach addresses the challenge of representing uncertainty in deep learning without compromising computational performance or test accuracy. The study delved further into exploring the properties of dropout uncertainty across various network architectures and non-linearities. Tasks such as extrapolation, interpolation, and classification were evaluated to understand the impact of model uncertainty. Notably, the researchers found that incorporating model uncertainty was particularly important for classification tasks, as exemplified by using the MNIST dataset. Moreover, the research team showcased how model uncertainty could be integrated into a Bayesian pipeline with deep reinforcement learning serving as a concrete application example. By demonstrating the practical implications of incorporating dropout as a Bayesian approximation for modeling uncertainty in deep learning, this study opens up new avenues for enhancing predictive accuracy and decision-making processes in machine learning applications.