# Controlled Training Data Generation with Diffusion Models

**Authors:** Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Esmaeil Akhoondi, Amir Zamir  
**arXiv:** [2403.15309](https://arxiv.org/abs/2403.15309)  
**Date:** 2024-03-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.15309v1/

## Key points

- Authors introduce a novel method for controlling a text-to-image generative model to generate training data tailored for supervised learning
- Proposed approach utilizes an automated closed-loop system with two feedback mechanisms to improve the generation process
- First feedback mechanism identifies adversarial prompts that lead to image generations maximizing the model loss
- Second feedback mechanism guides the generation process towards a specific target distribution
- Combined approach termed Guided Adversarial Prompts
- Evaluations demonstrate effectiveness of proposed feedback mechanisms compared to conventional open-loop methods in enhancing supervised learning outcomes

## Summary

In their work titled "Controlled Training Data Generation with Diffusion Models," authors Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev, Ruchira Ray, Pooya Esmaeil Akhoondi, and Amir Zamir introduce a novel method for controlling a text-to-image generative model to generate training data tailored for supervised learning. The proposed approach departs from traditional open-loop methods that rely on predefined prompts generated by language models or human expertise. Instead, the authors propose an automated closed-loop system with two feedback mechanisms to improve the generation process. The first feedback mechanism utilizes a supervised model to identify adversarial prompts that lead to image generations maximizing the model loss. These adversarial prompts result in diverse data influenced by the model but may not align efficiently with the target distribution. To address this limitation, the authors introduce a second feedback mechanism aimed at guiding the generation process towards a specific target distribution. This combined approach is termed Guided Adversarial Prompts. Through evaluations across various tasks, datasets, and architectures encompassing different types of distribution shifts such as spuriously correlated data and unseen domains, the authors demonstrate the effectiveness of their proposed feedback mechanisms compared to conventional open-loop methods. Their research showcases how controlled training data generation using diffusion models can enhance supervised learning outcomes by producing more relevant and targeted datasets for training machine learning models.