# Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning

**Authors:** Ting Chen, Ruixiang Zhang, Geoffrey Hinton  
**arXiv:** [2208.04202](https://arxiv.org/abs/2208.04202)  
**Date:** 2022-08-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2208.04202v2/

## Key points

- The paper presents a novel approach called Bit Diffusion for generating discrete data using continuous state and continuous time diffusion models.
- Discrete data is represented as binary bits and trained on a continuous diffusion model to model these bits as real numbers (analog bits).
- Samples are generated by first generating analog bits, which are then thresholded to obtain the bits representing the discrete variables.
- Two techniques, Self-Conditioning and Asymmetric Time Intervals, are proposed to improve the quality of generated samples.
- Outperforms the best autoregressive model in terms of sample quality and efficiency on CIFAR-10 and ImageNet-64x64 datasets for image generation.
- Achieves competitive results compared to autoregressive models for image captioning on the MS-COCO dataset.
- Authors: Ting Chen, Ruixiang Zhang, Geoffrey Hinton.
- Accepted at ICLR'23 under categories: computer vision (cs.CV), artificial intelligence (cs.AI), computational linguistics (cs.CL), machine learning (cs.LG).

## Summary

The paper titled "Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning" presents a novel approach called Bit Diffusion for generating discrete data using continuous state and continuous time diffusion models. The main idea behind this approach is to represent discrete data as binary bits and then train a continuous diffusion model to model these bits as real numbers, which are referred to as analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits representing the discrete variables. The authors also propose two techniques, namely Self-Conditioning and Asymmetric Time Intervals, which significantly improve the quality of generated samples. In terms of discrete image generation, it outperforms the best autoregressive model in both sample quality (measured by FID) and efficiency on CIFAR-10 (3K discrete 8-bit tokens) and ImageNet-64x64 (12K discrete 8-bit tokens) datasets. For image captioning on the MS-COCO dataset, it achieves competitive results compared to autoregressive models. The authors of this paper are Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. The paper was accepted at ICLR'23 and falls under the categories of computer vision (cs.CV), artificial intelligence (cs.AI), computational linguistics (cs.CL), and machine learning (cs.LG).