In the realm of virtual try-on research, the primary focus has been on catering to the fashion industry's needs by creating images that showcase garments on studio models in a cost-effective manner. However, there is a growing recognition that virtual try-on technology should extend beyond this narrow scope to enable customers to visualize clothing items on themselves using their own everyday photos, a concept known as in-the-wild try-on. The challenge lies in the fact that existing methods, which excel in generating realistic results for studio settings, struggle when applied to in-the-wild scenarios due to their reliance on paired data (garment images matched with images of individuals wearing those garments) for training. While obtaining such paired data is relatively straightforward for studio environments through shopping websites, it proves to be a significant hurdle in capturing diverse and uncontrolled real-world scenes. To address this gap, a team of researchers led by Aiyu Cui, Jay Mahajan, Viraj Shah, Preeti Gomathinayagam, Chang Liu, and Svetlana Lazebnik introduces a groundbreaking approach. They first establish the StreetTryOn benchmark aimed at supporting in-the-wild virtual try-on applications. Subsequently, they propose an innovative method that learns virtual try-on directly from a collection of unpaired person images taken in natural settings without the need for paired data. This novel technique tackles the unique challenges posed by in-the-wild scenarios by employing a DensePose warping correction method coupled with diffusion-based conditional inpainting. Through extensive experiments, the team demonstrates not only competitive performance for traditional studio try-on tasks but also state-of-the-art results for street try-on and cross-domain try-on tasks. In essence, this research represents a significant step forward in expanding the capabilities of virtual try-on technology beyond controlled studio environments towards more versatile and practical applications that cater to real-world scenarios where customers can seamlessly visualize clothing items on themselves using their own photographs.
- Virtual try-on research has primarily focused on showcasing garments on studio models for the fashion industry in a cost-effective manner.
- There is a growing recognition of the need to extend virtual try-on technology to enable customers to visualize clothing items on themselves using their own everyday photos (in-the-wild try-on).
- Existing methods struggle with in-the-wild scenarios due to the reliance on paired data, which is more readily available for studio settings compared to diverse real-world scenes.
- A team of researchers led by Aiyu Cui, Jay Mahajan, Viraj Shah, Preeti Gomathinayagam, Chang Liu, and Svetlana Lazebnik introduced a groundbreaking approach:
- Established the StreetTryOn benchmark to support in-the-wild virtual try-on applications.
- Proposed an innovative method that learns virtual try-on directly from unpaired person images taken in natural settings without paired data.
- Utilized DensePose warping correction and diffusion-based conditional inpainting techniques to address challenges posed by in-the-wild scenarios.
- Through extensive experiments, the team demonstrated competitive performance for traditional studio try-on tasks and state-of-the-art results for street try-on and cross-domain try-on tasks.
Summary- Virtual try-on research has focused on showing clothes on models in a cost-effective way for the fashion industry.
- Now, there is a need to let customers see how clothes look on them using their own photos (in-the-wild try-on).
- Current methods struggle with real-life scenarios because they rely on specific data that is easier to get in studios than in diverse settings.
- A team of researchers introduced a new way to do virtual try-on outside studios called StreetTryOn.
- They used innovative techniques to learn from unpaired images and achieved great results for different try-on tasks.
Definitions- Virtual try-on: Trying on clothes virtually using technology.
- In-the-wild: Real-life, everyday situations.
- Paired data: Matching sets of information used in algorithms.
- Benchmark: Standard or reference point for comparison.
- Unpaired images: Pictures not specifically matched or related.
In-the-Wild Virtual Try-On: Expanding the Capabilities of Virtual Try-On Technology
In today's fast-paced world, technology has revolutionized the way we shop for clothing. With the rise of e-commerce and online shopping, customers are no longer limited to physical stores and can browse through a wide range of options from the comfort of their own homes. However, one major drawback of online shopping is not being able to try on clothes before making a purchase. This is where virtual try-on technology comes in.
Virtual try-on allows customers to visualize how clothing items will look on them without physically trying them on. This technology has been primarily focused on catering to the needs of the fashion industry by creating images that showcase garments on studio models in a cost-effective manner. But with changing consumer demands, there is a growing recognition that virtual try-on should extend beyond this narrow scope and enable customers to visualize clothing items on themselves using their own everyday photos - a concept known as in-the-wild try-on.
However, implementing virtual try-on for in-the-wild scenarios poses unique challenges due to its reliance on paired data (garment images matched with images of individuals wearing those garments) for training. While obtaining such paired data is relatively straightforward for studio environments through shopping websites, it proves to be a significant hurdle in capturing diverse and uncontrolled real-world scenes.
To address this gap, Aiyu Cui and his team at Adobe Research have introduced an innovative approach that expands the capabilities of virtual try-on technology towards more versatile and practical applications that cater to real-world scenarios.
The StreetTryOn Benchmark
The first step taken by Cui et al. was establishing the StreetTryOn benchmark aimed at supporting in-the-wild virtual try-on applications. The benchmark consists of two datasets - StreetClothes dataset containing street-style garment images collected from various sources and Fashionista dataset consisting of street-style images with corresponding clothing labels. This benchmark serves as a standardized evaluation platform for in-the-wild virtual try-on methods.
An Innovative Method
The team's proposed method learns virtual try-on directly from a collection of unpaired person images taken in natural settings without the need for paired data. This novel technique tackles the unique challenges posed by in-the-wild scenarios by employing a DensePose warping correction method coupled with diffusion-based conditional inpainting.
DensePose is a state-of-the-art pose estimation algorithm that maps image pixels to 3D surface coordinates, providing detailed body part segmentation and pose estimation. The team uses this information to warp the garment image onto the person's body, correcting any distortions caused by different poses or camera angles.
Diffusion-based conditional inpainting is then used to fill in missing parts of the warped garment image, resulting in a realistic and seamless virtual try-on result. This approach eliminates the need for paired data and allows for more diverse and uncontrolled real-world scenes to be used as training data.
Impressive Results
Through extensive experiments on both traditional studio try-on tasks and in-the-wild scenarios using the StreetTryOn benchmark, Cui et al.'s method demonstrates competitive performance compared to existing methods. It also achieves state-of-the-art results for cross-domain try-on tasks where garments from one dataset are applied to people from another dataset.
This research represents a significant step forward in expanding the capabilities of virtual try-on technology beyond controlled studio environments towards more versatile and practical applications that cater to real-world scenarios. With this innovative approach, customers can seamlessly visualize clothing items on themselves using their own photographs, making online shopping even more convenient and personalized.
In conclusion, Cui et al.'s research paper highlights the potential of virtual try-on technology beyond its traditional use in studio settings. By introducing an innovative method that eliminates the need for paired data and establishing a benchmark for in-the-wild virtual try-on, this research opens up new possibilities for the fashion industry and online shopping. As technology continues to advance, we can expect to see even more impressive applications of virtual try-on in the future.
