# How much is a noisy image worth? Data Scaling Laws for Ambient Diffusion

**Authors:** Giannis Daras, Yeshwanth Cherapanamjeri, Constantinos Daskalakis  
**arXiv:** [2411.02780](https://arxiv.org/abs/2411.02780)  
**Date:** 2024-11-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.02780v1/

## Key points

- Availability of large-scale, high-quality datasets is crucial for the success of generative models like Image Diffusion Models (DMs) and Large Language Models (LLMs)
- Obtaining clean data can be challenging in fields such as Magnetic Resonance Imaging (MRI) and black-hole imaging due to time limitations or physical impossibilities
- Scarcity of high-quality data has led to exploration of training generative models using corrupted data like blurry or noisy images
- Study involving over 80 models showed that a combination of a small set of clean data along with a larger set of highly noisy data can match performance of models trained solely on clean datasets
- Theoretical evidence suggests that incorporating clean samples into training can significantly reduce sample size requirements for noisy data, leading to enhanced model performance

## Summary

, , , , 

In the realm of generative models, the success of models like Image Diffusion Models (DMs) and Large Language Models (LLMs) hinges on the availability of large-scale, high-quality datasets. However, obtaining clean data can be challenging in certain fields such as Magnetic Resonance Imaging (MRI) and black-hole imaging due to constraints like time limitations or physical impossibilities. Even for general domain image datasets, creating a copyright-free large-scale dataset can be a costly and complex endeavor. This scarcity of high-quality data has led to the exploration of training generative models using corrupted data, such as blurry or noisy images. Recent frameworks have emerged to address this challenge by training generative models with solely corrupted data. However, these ambient diffusion models often fall short in performance compared to models trained on clean data. To delve deeper into this phenomenon, a study was conducted involving over 80 models trained on datasets with varying levels of corruption across different sample sizes ranging from 30,000 to approximately 1.3 million samples. The results revealed that achieving the same level of performance as models trained on clean data solely with noisy data is unattainable at these sample sizes. Nonetheless, a combination of a small set of clean data (e.g., 10% of the total dataset) along with a larger set of highly noisy data proved sufficient to match the performance of models trained solely on similar-sized clean datasets. This hybrid approach even enabled near state-of-the-art performance. The study also provided theoretical evidence through the development of novel sample complexity bounds for learning from Gaussian Mixtures with varying variances. The theoretical model indicated that for sufficiently large datasets, the marginal utility of a noisy sample diminishes exponentially compared to that of a clean sample. By incorporating a small subset of clean samples into training, significant reductions in sample size requirements for noisy data were observed – aligning with the experimental findings. Overall, this research sheds light on the value and impact of incorporating both clean and noisy data in training generative models and highlights how strategic combinations can lead to enhanced model performance even when faced with limited access to pristine datasets.