DreamFusion: Text-to-3D using 2D Diffusion

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation

   Jinbo Wu, Xiaobo Gao, Xing Liu et al. · 2023 · IEEE Workshop/Winter Conference on Applications of Computer Vision · cited 31 times · Not summarized yet

   Abstract

   In this paper, we study Text-to-3D content generation leveraging 2D diffusion priors to enhance the quality and detail of the generated 3D models. Recent progress [11] in text-to-3D has shown that employing high-resolution (e.g., 512 × 512) renderings can lead to the production of high-quality 3D models using latent diffusion priors. To enable rendering at even higher resolutions, which has the potential to further augment the quality and detail of the models, we propose a novel approach that combines multiple noise estimation processes with a pretrained 2D diffusion prior. Distinct from the Bar-Tal et al.s’ study which binds multiple denoised results [1] to generate images from texts, our a…

2. Very close Cites this paper Not relevant

   Magic3D: High-Resolution Text-to-3D Content Creation

   Chen-Hsuan Lin, Jun Gao, Luming Tang et al. · 2022 · Computer Vision and Pattern Recognition · cited 1,751 times · AI summary available

   Abstract

   DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a texture…

3. Very close Not relevant

   Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation

   Yu-Ze He, Yushi Bai, M. Lin et al. · 2023 · Computer Aided Geometric Design · cited 7 times · Not summarized yet

   Abstract

   By lifting the pre-trained 2D diffusion models into Neural Radiance Fields (NeRFs), text-to-3D generation methods have made great progress. Many state-of-the-art approaches usually apply score distillation sampling (SDS) to optimize the NeRF representations, which supervises the NeRF optimization with pre-trained text-conditioned 2D diffusion models such as Imagen. However, the supervision signal provided by such pre-trained diffusion models only depends on text prompts and does not constrain the multi-view consistency. To inject the cross-view consistency into diffusion priors, some recent works finetune the 2D diffusion model with multi-view data, but still lack fine-grained view coherence…

4. Very close Not relevant

   3D-CLFusion: Fast Text-to-3D Rendering with Contrastive Latent Diffusion

   Yu-Jhe Li, Tao Xu, Ji Hou et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 9 times · Not summarized yet

   Abstract

   We tackle the task of text-to-3D creation with pre-trained latent-based NeRFs (NeRFs that generate 3D objects given input latent code). Recent works such as DreamFusion and Magic3D have shown great success in generating 3D content using NeRFs and text prompts, but the current approach of optimizing a NeRF for every text prompt is 1) extremely time-consuming and 2) often leads to low-resolution outputs. To address these challenges, we propose a novel method named 3D-CLFusion which leverages the pre-trained latent-based NeRFs and performs fast 3D content creation in less than a minute. In particular, we introduce a latent diffusion prior network for learning the w latent from the input CLIP te…

5. Very close Not relevant

   Diverse and Stable 2D Diffusion Guided Text to 3D Generation with Noise Recalibration

   Xiaofeng Yang, Fa-Yao Liu, Yi Xu et al. · 2024 · AAAI Conference on Artificial Intelligence · cited 7 times · Not summarized yet

   Abstract

   In recent years, following the success of text guided image generation, text guided 3D generation has gained increasing attention among researchers. Dreamfusion is a notable approach that enhances generation quality by utilizing 2D text guided diffusion models and introducing SDS loss, a technique for distilling 2D diffusion model information to train 3D models. However, the SDS loss has two major limitations that hinder its effectiveness. Firstly, when given a text prompt, the SDS loss struggles to produce diverse content. Secondly, during training, SDS loss may cause the generated content to overfit and collapse, limiting the model's ability to learn intricate texture details. To overcome …

6. Very close Not relevant

   Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Data

   Zhi-Yuan Ma, Xinyue Liang, Rong-Yuan Wu et al. · 2025 · Computer Vision and Pattern Recognition · cited 8 times · Not summarized yet

   Abstract

   It is highly desirable to obtain a model that can generate high-quality 3D meshes from text prompts in just seconds. While recent attempts have adapted pre-trained text-to-image diffusion models, such as Stable Diffusion (SD), into generators of 3D representations (e.g., Triplane), they often suffer from poor quality due to the lack of sufficient high-quality 3D training data. Aiming at overcoming the data shortage, we propose a novel training scheme, termed as Progressive Rendering Distillation (PRD), eliminating the need for 3D ground-truths by distilling multi-view diffusion models and adapting SD into a native 3D generator. In each iteration of training, PRD uses the U-Net to progressive…

7. Very close Not relevant

   Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models

   Jia-Le Xu, Xintao Wang, Wei-Hao Cheng et al. · 2022 · Computer Vision and Pattern Recognition · cited 211 times · Not summarized yet

   Abstract

   Recent CLIP-guided 3D optimization methods, such as DreamFields [19] and PureCLIPNeRF [24], have achieved impressive results in zero-shot text-to-3D synthesis. However, due to scratch training and random initialization without prior knowledge, these methods often fail to generate accurate and faithful 3D structures that conform to the input text. In this paper, we make the first attempt to introduce explicit 3D shape priors into the CLIP-guided 3D optimization process. Specifically, we first generate a high-quality 3D shape from the input text in the text-to-shape stage as a 3D shape prior. We then use it as the initialization of a neural radiance field and optimize it with the full prompt. …

8. Very close Not relevant

   Creating High-Quality 3D Content by Bridging the Gap between Text-to-2D and Text-to-3D Generation

   Yi-Wei Ma, Yijun Fan, Jia-Yi Ji et al. · 2024 · ACM Trans. Multim. Comput. Commun. Appl. · cited 7 times · Not summarized yet

   Abstract

   In recent times, automatic text-to-3D content creation has made significant progress, driven by the development of pretrained 2D diffusion models. Existing text-to-3D methods typically optimize the 3D representation to ensure that the rendered image aligns well with the given text, as evaluated by the pretrained 2D diffusion model. Nevertheless, a substantial domain gap exists between 2D images and 3D assets, primarily attributed to variations in camera-related attributes and the exclusive presence of foreground objects. Consequently, employing 2D diffusion models directly for optimizing 3D representations may lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a novel …

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Denoising Diffusion Probabilistic Models

   Jonathan Ho, Ajay Jain, Pieter Abbeel · 2020 · Machine Learning · cited 37,366 times · AI summary available

   Abstract

   We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar…

2. Close Used in depth by this paper A method it uses Not relevant

   Zero-Shot Text-Guided Object Generation with Dream Fields

   Ajay Jain, B. Mildenhall, J. Barron et al. · 2021 · Computer Vision and Pattern Recognition · cited 683 times · Not summarized yet

   Abstract

   We combine neural rendering with multi-modal image and text representations to synthesize diverse 3D objects solely from natural language descriptions. Our method, Dream Fields, can generate the geometry and color of a wide range of objects without 3D supervision. Due to the scarcity of diverse, captioned 3D data, prior methods only generate objectsfrom a handful of categories, such as ShapeNet. Instead, we guide generation with image-text models pre-trained on large datasets of captioned images from the web. Our method optimizes a Neural Radiance Field from many camera views so that rendered images score highly with a target caption according to a pre-trained CLIP model. To improve fidelity…

3. Close A method it uses Not relevant

   GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

   Alex Nichol, Prafulla Dhariwal, Aditya Ramesh et al. · 2021 · Computer Vision and Pattern Recognition · cited 5,172 times · AI summary available

   Abstract

   Diffusion models have recently been shown to generate high-quality synthetic images, especially when paired with a guidance technique to trade off diversity for fidelity. We explore diffusion models for the problem of text-conditional image synthesis and compare two different guidance strategies: CLIP guidance and classifier-free guidance. We find that the latter is preferred by human evaluators for both photorealism and caption similarity, and often produces photorealistic samples. Samples from a 3.5 billion parameter text-conditional diffusion model using classifier-free guidance are favored by human evaluators to those from DALL-E, even when the latter uses expensive CLIP reranking. Addit…

4. Close A method it uses Not relevant

   Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

   Chitwan Saharia, William Chan, Saurabh Saxena et al. · 2022 · Computer Vision and Pattern Recognition · cited 9,191 times · AI summary available

   Abstract

   We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 …

5. Close Not relevant

   GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

   K. Schwarz, Yiyi Liao, Michael Niemeyer et al. · 2020 · Neural Information Processing Systems · cited 1,005 times · Not summarized yet

   Abstract

   While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthes…

Papers that cite it

1. Very close Not relevant

   RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

   Tengfei Wang, Bo Zhang, Ting Zhang et al. · 2023 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 210 times · Not summarized yet

   Abstract

   This paper presents a 3D diffusion model that automatically generates 3D digital avatars represented as neural radiance fields (NeRFs). A significant challenge for 3D diffusion is that the memory and processing costs are prohibitive for producing high-quality results with rich details. To tackle this problem, we propose the roll-out diffusion network (RODIN), which takes a 3D NeRF model represented as multiple 2D feature maps and rolls out them onto a single 2D feature plane within which we perform 3D-aware diffusion. The RODIN model brings much-needed computational efficiency while preserving the integrity of 3D diffusion by using 3D-aware convolution that attends to projected features in t…

2. Close Not relevant

   Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures

   Gal Metzer, Elad Richardson, Or Patashnik et al. · 2023 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 336 times · Not summarized yet

   Abstract

   Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resul…

3. Close Not relevant

   Wonder3D: Single Image to 3D Using Cross-Domain Diffusion

   Xiaoxiao Long, Yuan–Chen Guo, Cheng Chung Lin et al. · 2024 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 384 times · Not summarized yet

   Abstract

   In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works di-rectly produce 3D information via fast network inferences, but their results are often of low quality and lack geometric details. To holistically improve the quality, consistency, and efficiency of single-view reconstruction tasks, we pro-pose a cross-domain diffusion model that generates multi-view normal …

4. Close Uses this paper in depth Not relevant

   Anchoring and Steering Diffusion: Enhancing the Faithfulness of Text-to-Image Generation at Inference Time

   Xinyi Wang, Yuyang Huang, Yalin Su et al. · 2026 · European Conference on Computer Vision · Not summarized yet

   Abstract

   While text-to-image diffusion models achieve impressive visual quality, they frequently struggle to maintain precise alignment with complex compositional prompts. An effective strategy is to improve the inference process of diffusion models, thereby better leveraging their pretrained priors to address misalignment. Existing training-free methods can be divided into two categories. The first category focuses on improving the randomly sampled initial noise, either performing costly search over noise pools or manipulating sampled noise without ensuring reliable semantic injection. The second category focuses on improving the denoising trajectory, lacking explicit mechanisms to timely diagnose a…

5. Close Uses this paper in depth Not relevant

   Single Image to Textured 3D Object Generation in Frequency Domain: From Theory to Pipeline

   Qisen Wang, Yifan Zhao, Jia Li · 2026 · International Journal of Computer Vision · Not summarized yet

   Abstract

   Single-view 3D reconstruction, also known as image-to-3D, is a persistently challenging task due to the extreme lack of information. Recently, diffusion models pre-trained on large-scale datasets served as 2D priors are used to solve the ill-posed task but suffer from color deviation and view inconsistency, which can be curbed by using diffusion models fine-tuned with 3D annotated data served as 3D priors. However, 3D priors lack high-frequency details, which cannot be solved by direct complementation with 2D priors in spatial domain for introducing erroneous low-frequency 2D prior guidance. In this paper, we revisit the characteristics of different diffusion priors from the frequency perspe…