# Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

**Authors:** Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, Yan Yan  
**arXiv:** [2206.07771](https://arxiv.org/abs/2206.07771)  
**Date:** 2022-06-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.07771v2/

## Key points

- Diffusion probabilistic models (DPMs) have gained popularity in conditional generation
- Existing methods implicitly address the issue of input-output correspondence
- This work takes a different approach by explicitly enhancing the input-output connections through maximizing mutual information
- Introduces a novel loss function called Conditional Discrete Contrastive Diffusion (CDCD)
- Design two contrastive diffusion mechanisms to incorporate CDCD into the denoising process
- Connects diffusion training and contrastive learning with conventional variational objectives for the first time
- Demonstrates efficacy through evaluations on various multimodal conditional synthesis tasks
- Significant enhancement of input-output correspondence and higher or competitive synthesis quality across all evaluated tasks
- Reduces required diffusion steps by more than 35% on two benchmark datasets, leading to increased inference speed
- Presents a novel approach that enhances input-output connections in DPMs for conditional generation tasks

## Summary

Diffusion probabilistic models (DPMs) have gained popularity in the field of conditional generation due to their promising results and ability to support cross-modal synthesis. However, achieving high correspondence between the conditioning input and generated output remains a challenge. Most existing methods address this issue implicitly by incorporating the prior into the variational lower bound. In contrast, this work takes a different approach by explicitly enhancing the input-output connections through maximizing their mutual information. To accomplish this, the authors introduce a novel loss function called Conditional Discrete Contrastive Diffusion (CDCD) and design two contrastive diffusion mechanisms to effectively incorporate it into the denoising process. This is achieved by connecting diffusion training and contrastive learning with conventional variational objectives for the first time. The efficacy of this approach is demonstrated through evaluations on various multimodal conditional synthesis tasks including dance-to-music generation, text-to-image synthesis, and class-conditioned image synthesis. The proposed method significantly enhances the input-output correspondence and achieves higher or competitive general synthesis quality across all evaluated tasks. Additionally, it improves the convergence of diffusion models by reducing the number of required diffusion steps by more than 35% on two benchmark datasets, leading to a significant increase in inference speed. Overall, this work presents a novel approach that explicitly enhances input-output connections in DPMs for conditional generation tasks. The incorporation of CDCD loss and contrastive diffusion mechanisms not only improves synthesis quality but also accelerates model convergence, making it a valuable contribution to the field of cross-modal music and image generation.