# Black-Box Variational Inference Converges

**Authors:** Kyurae Kim, Kaiwen Wu, Jisu Oh, Yian Ma, Jacob R. Gardner  
**arXiv:** [2305.15349](https://arxiv.org/abs/2305.15349)  
**Date:** 2023-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.15349v1/

## Key points

- Authors present the first convergence guarantee for full black-box variational inference (BBVI)
- Previous investigations on BBVI focused on simplified versions that required algorithmic modifications
- The authors' setup does not require any such modifications
- Results hold for log-smooth posterior densities with and without strong log-concavity and the location-scale variational family
- Certain algorithm design choices commonly used in practice can lead to suboptimal convergence rates
- Solution proposed by running BBVI with proximal stochastic gradient descent
- Proximal SGD achieves the strongest known convergence rate guarantees
- Proximal SGD compared against other standard implementations of BBVI on large-scale Bayesian inference problems
- Effectiveness of proposed method demonstrated in practice
- Research provides valuable contributions to black-box variational inference by offering convergence guarantees without requiring simplifications or algorithmic modifications
- Findings highlight potential pitfalls in common algorithm design choices
- Solution offered through proximal stochastic gradient descent achieves optimal convergence rates.

## Summary

In their paper titled "Black-Box Variational Inference Converges," authors Kyurae Kim, Kaiwen Wu, Jisu Oh, Yian Ma, and Jacob R. Gardner present the first convergence guarantee for full black-box variational inference (BBVI), also known as Monte Carlo variational inference. Previous investigations on BBVI focused on simplified versions that required algorithmic modifications such as bounded domain or support, optimization only for the scale, etc. However, the authors' setup does not require any such modifications. The results of this study hold for log-smooth posterior densities with and without strong log-concavity and the location-scale variational family. The analysis conducted by the authors reveals that certain algorithm design choices commonly used in practice can lead to suboptimal convergence rates. Fortunately, they propose a solution to address these limitations by running BBVI with proximal stochastic gradient descent. This approach fixes the suboptimal convergence rates and achieves the strongest known convergence rate guarantees. To evaluate their theoretical insight, they compare proximal SGD against other standard implementations of BBVI on large-scale Bayesian inference problems. By doing so, they demonstrate the effectiveness of their proposed method in practice. Overall, this research provides valuable contributions to black-box variational inference by offering convergence guarantees without requiring simplifications or algorithmic modifications. The findings highlight potential pitfalls in common algorithm design choices and offer a solution to achieve optimal convergence rates through proximal stochastic gradient descent.