# Improving Diffusion Models for Virtual Try-on

**Authors:** Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, Jinwoo Shin  
**arXiv:** [2403.05139](https://arxiv.org/abs/2403.05139)  
**Date:** 2024-03-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.05139v2/

## Key points

- Image-based virtual try-on technology focuses on rendering images of people wearing specific garments based on input images.
- Previous approaches used exemplar-based inpainting diffusion models to improve visual quality but struggled with maintaining garment identity and fidelity.
- The IDM-VTON model, developed by researchers led by Yisol Choi et al., introduces a novel diffusion model with two distinct modules for encoding garment semantics.
- The IDM-VTON model integrates high-level semantics into the cross-attention layer and low-level features into the self-attention layer, enhancing fidelity and authenticity in generated virtual try-on images.
- Detailed textual prompts for both garment and person images are provided to enrich the authenticity of visuals further.
- A customization method using pairs of person-garment images significantly improves fidelity and authenticity in real-world scenarios.
- Experimental results show that IDM-VTON outperforms previous methods in preserving garment details and producing authentic virtual try-on images of superior quality.
- The customization method of IDM-VTON is particularly effective in practical applications, representing a significant advancement in virtual try-on technology.

## Summary

In the realm of image-based virtual try-on technology, where an image of a person is rendered wearing a specific garment based on two input images depicting the person and the clothing item, there has been a significant focus on improving the naturalness and authenticity of the generated visuals. Previous approaches have utilized exemplar-based inpainting diffusion models to enhance visual quality but often struggled to maintain the identity and fidelity of the garments being worn. To address this challenge, a team of researchers led by Yisol Choi, Sangkyung Kwak, Kyungmin Lee, Hyungwon Choi, and Jinwoo Shin introduced a novel diffusion model known as IDM-VTON. The IDM-VTON model incorporates two distinct modules that encode the semantics of garment images in a unique way. Firstly, high-level semantics extracted from a visual encoder are integrated into the cross-attention layer of the base UNet in the diffusion model. Secondly, low-level features extracted from a parallel UNet are fused into the self-attention layer. This dual-module approach enhances the overall fidelity and authenticity of virtual try-on images generated by IDM-VTON. Moreover, detailed textual prompts for both garment and person images are provided to further enrich the authenticity of the generated visuals. The researchers also introduced a customization method that utilizes pairs of person-garment images to significantly improve fidelity and authenticity in real-world scenarios. Experimental results demonstrate that IDM-VTON surpasses previous methods—both diffusion-based and GAN-based—in preserving garment details and producing authentic virtual try-on images with superior quality both qualitatively and quantitatively. The effectiveness of IDM-VTON's customization method was particularly highlighted in practical applications. More information and visualizations related to this research can be found on their project page at https://idm-vton.github.io. Overall,this study represents a significant advancement in virtual try-on technology by enhancing garment fidelity while generating visually appealing and authentic virtual try-on experiences for users.