# Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget

**Authors:** Vikash Sehwag, Xianghao Kong, Jingtao Li, Michael Spranger, Lingjuan Lyu  
**arXiv:** [2407.15811](https://arxiv.org/abs/2407.15811)  
**Date:** 2024-07-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.15811v1/

## Key points

- Development of large-scale text-to-image (T2I) diffusion transformer models concentrated among actors with substantial computational resources
- Novel approach proposed to democratize training of large-scale T2I diffusion models by lowering cost and resource requirements
- Leveraging vision transformer-based latent diffusion models for enhanced performance and minimized computational overhead
- Key innovations include deferred masking strategy, mixture-of-experts layers, and utilization of synthetic images in micro-budget training
- Training a 1.16 billion parameter sparse transformer at an economical cost of $1,890 using only 37 million publicly available real and synthetic images
- Achieving impressive results with a 12.7 Frechet Inception Distance (FID) score in zero-shot generation on the COCO dataset
- Significantly lower cost compared to stable diffusion models and current state-of-the-art methods, promoting inclusivity within the field

## Summary

, , , , 

In the realm of generative AI, the development of large-scale text-to-image (T2I) diffusion transformer models has been predominantly concentrated among actors with substantial computational resources due to the high cost associated with training these models from scratch. While previous works have made strides in reducing computational costs compared to traditional methods, the barrier to entry remains high, requiring extensive training time and access to vast datasets. In response to this challenge, a novel approach is proposed in this work that aims to democratize the training of large-scale T2I diffusion models by significantly lowering the cost and resource requirements. The focus is on developing a low-cost end-to-end pipeline for competitive T2I diffusion models that achieve remarkable reductions in training costs without the need for billions of training images or proprietary datasets. Leveraging vision transformer-based latent diffusion models known for their simplified design and widespread adoption across recent large-scale generative models, the proposed approach aims to enhance performance while minimizing computational overhead. Key innovations include a deferred masking strategy that preprocesses image patches before randomly masking up to 75% during training, effectively reducing performance degradation associated with masking and surpassing traditional model downscaling techniques in cost reduction. Additionally, incorporating state-of-the-art improvements such as mixture-of-experts layers and utilizing synthetic images in micro-budget training further enhances model performance. Notably, by leveraging only 37 million publicly available real and synthetic images, a 1.16 billion parameter sparse transformer is trained at an economical cost of $1,890. This model achieves impressive results with a 12.7 Frechet Inception Distance (FID) score in zero-shot generation on the COCO dataset. Comparatively, this approach incurs 118 times lower cost than stable diffusion models and 14 times lower cost than current state-of-the-art methods costing $28,400. Ultimately, this work aims to release an accessible end-to-end training pipeline that empowers researchers and developers to train large-scale diffusion models on micro-budgets efficiently. By bridging the gap between computational resources and model performance, this innovative approach opens up new possibilities for advancing generative AI research while promoting inclusivity within the field.