# Generative Pretraining in Multimodality

**Authors:** Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, Xinlong Wang  
**arXiv:** [2307.05222](https://arxiv.org/abs/2307.05222)  
**Date:** 2023-07-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.05222v1/

## Key points

- Emu is a powerful Transformer-based model for generating images and texts in a multimodal context.
- It can process single-modality or multimodal data inputs, such as interleaved image, text, and video.
- Emu encodes visual signals into embeddings and combines them with text tokens to form an interleaved input sequence.
- It is trained end-to-end to classify the next text token or regress the next visual embedding in the multimodal sequence.
- Emu can explore diverse pretraining data sources at scale, including videos, webpages, image-text pairs, and video-text pairs.
- It serves as a generalist multimodal interface for both image-to-text and text-to-image tasks.
- Emu outperforms state-of-the-art large multimodal models across various zero shot/few shot tasks like image captioning and visual question answering.
- It can also serve as a multimodal assistant via instruction tuning with impressive performance.
- Qualitative evaluations demonstrate Emu's impressive capabilities that cannot be solely evaluated based on quantitative benchmarks.
- Emu proves to be an advanced solution for multimodal generation tasks.

## Summary

Emu is a powerful Transformer-based model that excels in generating images and texts within a multimodal context. It can seamlessly process any single-modality or multimodal data input, such as interleaved image, text, and video, through a one-model-for-all autoregressive training process. The model encodes visual signals into embeddings and combines them with text tokens to form an interleaved input sequence. Emu is then trained end-to-end with the objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality allows Emu to explore diverse pretraining data sources at scale, including videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. As a result, Emu serves as a generalist multimodal interface for both image-to-text and text-to-image tasks, enabling in-context image and text generation. In terms of performance, Emu outperforms state-of-the art large multimodal models across various zero shot/few shot tasks such as image captioning, visual question answering, video question answering and text to image generation. Its extended capabilities include serving as a multimodal assistant via instruction tuning with impressive performance. The authors conducted qualitative evaluations of Emu to showcase its impressive capabilities that cannot be evaluated solely based on quantitative benchmarks. These real world applications demonstrate Emu's effectiveness in generating high quality outputs. In summary, Emu proves to be an advanced solution for multimodal generation tasks.