# Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning

**Authors:** Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, Ishan Misra  
**arXiv:** [2311.10709](https://arxiv.org/abs/2311.10709)  
**Date:** 2023-11-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.10709v1/

## Key points

- Emu Video is a state-of-the-art text-to-video generation model
- It introduces a novel factorization approach to the generation process
- The model divides the generation into two distinct steps: image generation and video generation
- This factorization allows for more efficient and effective video generation compared to previous methods
- Adjusted noise schedules for diffusion help in generating high-quality videos with high resolution directly
- Multi-stage training techniques further enhance the quality of the generated videos
- In human evaluations, Emu Video outperforms all prior work in terms of video quality
- It achieves an impressive preference over Google's Imagen Video, Nvidia's PYOCO, Meta's Make-A-Video, RunwayML's Gen2, and Pika Labs
- Emu Video has the ability to animate images based on user-provided text prompts
- Animations produced by Emu Video are preferred 96% more than those produced by previous methods
- Overall, Emu Video represents a significant advancement in text-to-video generation technology

## Summary

Emu Video is a state-of-the-art text-to-video generation model that introduces a novel factorization approach to the generation process. The model divides the generation into two distinct steps: first, it generates an image based on the given text, and then it generates a video by conditioning on both the text and the generated image. This factorization allows for more efficient and effective video generation compared to previous methods that relied on deep cascades of models. The authors of Emu Video address critical design decisions that significantly contribute to its success. They propose adjusted noise schedules for diffusion which helps in generating high-quality videos with high resolution directly. Additionally, they implement multi-stage training techniques that further enhance the quality of the generated videos. In human evaluations, Emu Video outperforms all prior work in terms of video quality. It achieves an impressive 81% preference over Google's Imagen Video, 90% preference over Nvidia's PYOCO, and an outstanding 96% preference over Meta's Make-A-Video. Moreover, commercial solutions such as RunwayML's Gen2 and Pika Labs are also surpassed by Emu Video. One notable advantage of Emu Video is its ability to animate images based on user-provided text prompts. The factorizing approach naturally lends itself to this task resulting in animations that are preferred 96% more than those produced by previous methods. Overall, Emu Video represents a significant advancement in text-to-video generation technology. Its factorization approach coupled with adjusted noise schedules and multi-stage training enables the direct production of high quality videos without relying on complex cascades of models. The model's superior performance in human evaluations highlights its effectiveness compared to both academic and commercial alternatives.