# High-Resolution Image Synthesis with Latent Diffusion Models

**Authors:** Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer  
**arXiv:** [2112.10752](https://arxiv.org/abs/2112.10752)  
**Date:** 2021-12-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.10752v2/

## Key points

- Paper introduces a novel approach to image synthesis using diffusion models (DMs) in the latent space of pretrained autoencoders
- DMs decompose the image formation process into sequential denoising autoencoder applications, achieving state-of-the-art results in image synthesis
- DMs offer a guiding mechanism for controlling image generation without retraining
- Applying DMs in the latent space of powerful pretrained autoencoders balances complexity reduction and detail preservation, resulting in improved visual fidelity
- Cross-attention layers are introduced to enhance diffusion models, enabling accommodation of general conditioning inputs and high-resolution synthesis in a convolutional manner
- Latent diffusion models (LDMs) achieve new state-of-the-art performance for tasks such as image inpainting, unconditional image generation, semantic scene synthesis, and super-resolution
- LDMs significantly reduce computational requirements while maintaining competitive performance compared to pixel-based DMs
- Code for implementing these models is available at https://github.com/CompVis/latent-diffusion

## Summary

The paper titled "High-Resolution Image Synthesis with Latent Diffusion Models" introduces a novel approach to image synthesis using diffusion models (DMs) applied in the latent space of pretrained autoencoders. DMs have been successful in achieving state-of-the-art results in image synthesis by decomposing the image formation process into sequential denoising autoencoder applications. These models also offer a guiding mechanism for controlling the image generation process without requiring retraining. To address computational requirements and expensive inference due to sequential evaluations, the authors propose applying DMs in the latent space of powerful pretrained autoencoders. This approach allows them to strike a near-optimal balance between complexity reduction and detail preservation, resulting in improved visual fidelity. To further enhance the capabilities of diffusion models, the authors introduce cross-attention layers into the model architecture. This addition transforms diffusion models into powerful and flexible generators that can accommodate general conditioning inputs such as text or bounding boxes. Moreover, it enables high-resolution synthesis in a convolutional manner. The proposed method, called latent diffusion models (LDMs), achieves new state-of-the-art performance for various tasks including image inpainting, unconditional image generation, semantic scene synthesis, and super-resolution. Compared to pixel-based DMs, LDMs significantly reduce computational requirements while maintaining competitive performance. Overall, this paper presents an innovative approach to high-resolution image synthesis by leveraging latent diffusion models trained on pretrained autoencoders. The results demonstrate improved visual fidelity and reduced computational costs compared to existing methods. The code for implementing these models is available at https://github.com/CompVis/latent-diffusion .