# Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors

**Authors:** Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, Bernard Ghanem  
**arXiv:** [2306.17843](https://arxiv.org/abs/2306.17843)  
**Date:** 2023-06-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.17843v2/

## Key points

- Magic123 is a cutting-edge two-stage approach for generating high-quality, textured 3D meshes from a single unposed image.
- Incorporates both 2D and 3D priors to surpass previous studies and achieve state-of-the-art results in image-to-3D reconstruction.
- Introduction of a trade-off parameter between the 2D and 3D priors enables control over the balance between exploration and exploitation of the generated geometry, leading to more realistic and detailed outputs.
- First stage involves optimizing a neural radiance field to create a coarse geometry, setting the foundation for the subsequent stage where a memory-efficient differentiable mesh representation produces high-resolution meshes with visually appealing textures.
- Learning of 3D content through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors throughout both stages.
- Limitations include assumptions that the reference image is taken from the front view, potential errors due to dependency on preprocessed segmentation and monocular depth estimation models, as well as issues with over-saturation in textures at higher resolutions in the second stage due to SDS loss.
- Despite limitations, Magic123 represents significant advancement in single image 3D reconstruction, demonstrating superiority over existing techniques in realism and level of detail through extensive experiments on real-world images and synthetic benchmarks.
- Availability of code, models, and generated assets on GitHub enhances accessibility for researchers interested in leveraging or building upon this methodology.

## Summary

Magic123 is a cutting-edge two-stage approach for generating high-quality, textured 3D meshes from a single unposed image. By incorporating both 2D and 3D priors, this method surpasses previous studies and achieves state-of-the-art results in image-to-3D reconstruction. The introduction of a trade-off parameter between the 2D and 3D priors enables control over the balance between exploration and exploitation of the generated geometry, leading to more realistic and detailed outputs. In the first stage of Magic123, a neural radiance field is optimized to create a coarse geometry. This initial step sets the foundation for the subsequent stage, where a memory-efficient differentiable mesh representation is employed to produce a high-resolution mesh with visually appealing textures. Throughout both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. However, it's important to note some limitations of this approach. One constraint is that Magic123 assumes the reference image is taken from the front view, which can lead to poor geometry when this assumption does not hold true (e.g., images taken from an alternative angle). Additionally, dependency on preprocessed segmentation and monocular depth estimation models may introduce errors that impact overall generation quality. Furthermore, there may be issues with over-saturation in textures due to the usage of SDS loss, particularly in the second stage at higher resolutions. Despite these limitations, Magic123 represents a significant advancement in single image 3D reconstruction. Extensive experiments on real-world images and synthetic benchmarks demonstrate its superiority over existing techniques in terms of realism and level of detail. By narrowing the gap between human abilities in 3D reasoning and machine capabilities, this work paves the way for future advancements in this field. The availability of code, models, and generated 3D assets on GitHub further enhances accessibility for researchers and practitioners interested in leveraging or building upon this innovative methodology. This research was supported by funding from KAUST Office of Sponsored Research through Visual Computing Center funding as well as support from SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI) along with contributions from KAUST Ibn Rushd Postdoc Fellowship program.