# InstructPix2Pix: Learning to Follow Image Editing Instructions

**Authors:** Tim Brooks, Aleksander Holynski, Alexei A. Efros  
**arXiv:** [2211.09800](https://arxiv.org/abs/2211.09800)  
**Date:** 2022-11-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.09800v1/

## Key points

- Proposed method: InstructPix2Pix, for editing images based on human instructions
- Model takes input image and written instruction, follows instructions to edit the image
- Training data generated by combining knowledge of pretrained language model (GPT-3) and text-to-image model (Stable Diffusion)
- Generalizes to real images and user-written instructions at inference time
- Performs edits in forward pass without per example fine-tuning or inversion, making it fast and efficient
- Uses Prompt-to-Prompt method to guarantee similar images for similar text prompts
- Compares with SDEdit, a baseline method that noise and denoise an input image with a new target prompt
- Enables editing from action-specific instructions rather than relying on labels or descriptions
- Uses off-the-shelf generative models for generating training data
- Shows compelling editing outcomes for various input images and written instructions
- Can achieve compounded edits by applying the model recurrently with different instructions
- Can produce multiple possible image edits for the same input image and instruction by varying latent noise
- Some failure cases where certain edits are not possible or undesired excessive changes occur

## Summary

. We propose a method for editing images based on human instructions. Our model, called InstructPix2Pix, takes an input image and a written instruction as input and follows these instructions to edit the image. To train our model, we generate a large dataset of image editing examples by combining the knowledge of two pretrained models - a language model (GPT-3) and a text-to-image model (Stable Diffusion). This allows us to generalize our model to real images and user-written instructions at inference time. Unlike other approaches, our model performs edits in the forward pass without requiring per example fine-tuning or inversion, making it fast and efficient. Previous works have used pretrained text-to-image diffusion models for image editing but they often lack the ability to guarantee similar images for similar text prompts. We address this issue by using a method called Prompt-to-Prompt, which assimilates generated images for similar text prompts, allowing isolated edits to be made. We also compare our approach with SDEdit, a baseline method that uses a pretrained model to noise and denoise an input image with a new target prompt. Our method differs from existing text-based image editing works in that it enables editing from instructions that specify the action to perform rather than relying on labels, captions or descriptions of input/output images. This allows users to provide precise and intuitive instructions in natural written text without needing additional information such as example images or constant visual content descriptions. To generate training data for our editing model, we use two off-the-shelf generative models - a language model and a text-to-image model. These generative models provide cheap and plentiful training data for downstream tasks. In terms of results, we show compelling editing outcomes for various input images and written instructions. By applying our model recurrently with different instructions, we can achieve compounded edits. Additionally, by varying the latent noise in our model, we can produce multiple possible image edits for the same input image and instruction. However, there are also some failure cases where our model is not capable of performing certain edits or may make undesired excessive changes to the image. It can also struggle with isolating specified objects or reorganizing/swapping objects.