# Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM

**Authors:** Chun-Liang Li, Siamak Ravanbakhsh, Barnabas Poczos  
**arXiv:** [1611.03879](https://arxiv.org/abs/1611.03879)  
**Date:** 2016-11-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1611.03879v1/

## Key points

- The paper explores an alternative member of the exponential family Restricted Boltzmann Machine (RBM) with leaky rectified linear units, called leaky RBM.
- The authors study the joint and marginal distributions of leaky RBM under different leakiness levels, connecting the model to truncated Gaussian distributions.
- They propose a simple yet efficient method for sampling from this model by annealing the leakiness rather than the energy.
- This serves as an alternative to annealing the temperature parameter and enables numerical estimation of likelihood that is more efficient and accurate than commonly used annealed importance sampling (AIS).
- The proposed sampling algorithm enjoys faster mixing property than contrastive divergence algorithm which benefits training without any additional computational cost.
- Their approach outperforms AIS in terms of both efficiency and accuracy on synthetic data sets.
- Their method can be applied to real-world datasets such as MNIST and CIFAR-10 with competitive performance compared to other state-of-the-art methods.

## Summary

The paper "Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM" by Chun-Liang Li, Siamak Ravanbakhsh, and Barnabas Poczos explores an alternative member of the exponential family Restricted Boltzmann Machine (RBM) with leaky rectified linear units, called leaky RBM. The authors study the joint and marginal distributions of leaky RBM under different leakiness levels, connecting the model to truncated Gaussian distributions. This connection leads to a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy. Specifically, they start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations. This serves as an alternative to annealing the temperature parameter and enables numerical estimation of likelihood that is more efficient and accurate than commonly used annealed importance sampling (AIS). The proposed sampling algorithm enjoys faster mixing property than contrastive divergence algorithm which benefits training without any additional computational cost. The authors demonstrate that their approach outperforms AIS in terms of both efficiency and accuracy on synthetic data sets. They also show that their method can be applied to real-world datasets such as MNIST and CIFAR-10 with competitive performance compared to other state-of-the-art methods. Overall, this paper presents a novel approach for sampling from exponential family RBMs with leaky rectified linear units using an annealing strategy based on decreasing leakiness levels rather than temperature parameters. The proposed method offers improved efficiency and accuracy compared to existing techniques like AIS while requiring no additional computational cost.