# PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

**Authors:** Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, Ari S. Morcos  
**arXiv:** [2308.03977](https://arxiv.org/abs/2308.03977)  
**Date:** 2023-08-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.03977v1/

## Key points

- Synthetic image datasets offer advantages for designing and evaluating deep neural networks
- Synthetic data allows for rendering an unlimited number of data samples with precise control over each scene
- It enables researchers to precisely control distribution shifts between training and testing for experimentation
- The use of synthetic image data is limited due to its lack of realism
- Most works rely on datasets of real images scraped from the internet, which have issues related to privacy, bias, copyright infringement, and limited control over object appearance
- PUG (Photorealistic Unreal Graphics) is a new generation of interactive environments proposed by the authors
- PUG utilizes the Unreal Engine to create realistic environments and datasets for representation learning research
- PUG provides both controllability and realism in generating synthetic data
- The authors aim to enable more rigorous evaluations of vision models through PUG environments
- The advancement of using PUG can overcome limitations associated with real image datasets while maintaining high levels of control over experimental variables

## Summary

In their paper titled "PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning," Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari S. Morcos discuss the advantages of using synthetic image datasets for designing and evaluating deep neural networks. They highlight that synthetic data allows for rendering an unlimited number of data samples with precise control over each scene resulting in granular ground truth labels and captions. Additionally, it enables researchers to precisely control distribution shifts between training and testing to isolate variables of interest for experimentation. Despite these benefits, the use of synthetic image data is still limited due to its lack of realism. As a result, most works rely on datasets of real images scraped from the internet which may have issues related to privacy, bias, copyright infringement and limited control over object appearance. To address this limitation and democratize the use of photorealistic synthetic data, the authors propose a new generation of interactive environments called PUG (Photorealistic Unreal Graphics). They utilize the Unreal Engine—a powerful game engine known in the entertainment industry—to create realistic environments and datasets for representation learning research. By leveraging PUG environments they aim to enable more rigorous evaluations of vision models. The authors demonstrate the potential of PUG by showcasing its ability to provide both controllability and realism in generating synthetic data. Their work opens up possibilities for researchers to overcome limitations associated with real image datasets while maintaining high levels of control over experimental variables. Ultimately this advancement can lead to improved design and evaluation processes for deep neural networks in various applications involving computer vision.