# Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

**Authors:** Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang  
**arXiv:** [2404.02905](https://arxiv.org/abs/2404.02905)  
**Date:** 2024-04-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.02905v1/

## Key points

- Visual AutoRegressive modeling (VAR) revolutionizes image learning and generation
- VAR predicts multi-scale token maps from coarse to fine scales, different from traditional autoregressive models
- VAR surpasses diffusion transformers in image generation tasks on ImageNet 256x256 benchmark
- VAR outperforms baseline autoregressive models in metrics like Frechet inception distance (FID) and inception score (IS), with around 20 times faster inference speed
- Empirical evidence shows VAR excels over Diffusion Transformers (DiT) in image quality, inference speed, data efficiency, and scalability
- Scaling up VAR models demonstrates power-law scaling laws similar to Large Language Models (LLMs)
- VAR showcases zero-shot generalization abilities in tasks like image in-painting, out-painting, and editing
- All models and codes related to VAR have been released for public access, facilitating further research in autoregressive modeling for visual generation

## Summary

In the realm of autoregressive modeling, a new paradigm called Visual AutoRegressive modeling (VAR) has emerged, revolutionizing the way images are learned and generated. Unlike traditional autoregressive models that predict tokens sequentially in a raster-scan order, VAR takes a novel approach by predicting multi-scale token maps from coarse to fine scales. This innovative methodology allows autoregressive transformers to quickly learn visual distributions and achieve superior generalization capabilities. One of the key achievements of VAR is its ability to surpass diffusion transformers in image generation tasks. On the ImageNet 256x256 benchmark, VAR significantly outperforms baseline autoregressive models by improving metrics such as Frechet inception distance (FID) and inception score (IS), while also achieving around 20 times faster inference speed. Empirical evidence confirms that VAR excels over Diffusion Transformers (DiT) across various dimensions including image quality, inference speed, data efficiency, and scalability. Furthermore, scaling up VAR models demonstrates clear power-law scaling laws similar to those observed in Large Language Models (LLMs), with linear correlation coefficients providing solid evidence of its effectiveness. Additionally, VAR showcases zero-shot generalization abilities in downstream tasks such as image in-painting, out-painting, and editing. These results indicate that VAR has successfully emulated two crucial properties of LLMs: Scaling Laws and zero-shot task generalization. To facilitate further exploration and research in the field of autoregressive modeling for visual generation and unified learning, all models and codes related to VAR have been released for public access. The introduction of VAR marks a significant advancement in autoregressive modeling for images, offering a promising avenue for future developments in this domain.