# VideoComposer: Compositional Video Synthesis with Motion Controllability

**Authors:** Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, Jingren Zhou  
**arXiv:** [2306.02018](https://arxiv.org/abs/2306.02018)  
**Date:** 2023-06-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.02018v1/

## Key points

- VideoComposer is a novel approach to controllable video synthesis
- It allows users to compose videos with specific conditions
- VideoComposer adopts the paradigm of compositional generation to address challenges associated with temporal dynamics and achieving higher inter-frame consistency
- It introduces textual, spatial, and temporal conditions as control signals for video composition
- Motion vectors from compressed videos are used as explicit control signals to guide the model in capturing temporal dynamics accurately
- The Spatio-Temporal Condition encoder (STC-encoder) effectively incorporates spatial and temporal relations of sequential inputs
- STC-encoder serves as a unified interface for better utilization of temporal conditions, leading to higher inter-frame consistency in synthesized videos
- VideoComposer can control spatial and temporal patterns within synthesized videos using various input forms such as text descriptions, sketch sequences, reference videos or hand-crafted motions
- Extensive experiments demonstrate that VideoComposer can generate visually appealing and coherent videos while adhering to user-specified conditions
- The code and models of VideoComposer will be publicly available at https://videocomposer.github.io for further exploration by researchers and practitioners.

## Summary

VideoComposer is a novel approach to controllable video synthesis that allows users to flexibly compose videos with specific conditions. To address the challenges associated with temporal dynamics and achieving higher inter-frame consistency, VideoComposer adopts the paradigm of compositional generation. It introduces textual, spatial, and temporal conditions as control signals for video composition and leverages motion vectors from compressed videos as explicit control signals to guide the model in capturing temporal dynamics accurately. The authors also propose a Spatio-Temporal Condition encoder (STC-encoder) that effectively incorporates spatial and temporal relations of sequential inputs. This encoder serves as a unified interface for better utilization of temporal conditions, leading to higher inter-frame consistency in synthesized videos. Extensive experiments demonstrate that VideoComposer can simultaneously control spatial and temporal patterns within synthesized videos using various input forms such as text descriptions, sketch sequences, reference videos or hand-crafted motions. The results showcase the ability of VideoComposer to generate visually appealing and coherent videos while adhering to user-specified conditions. The code and models of VideoComposer will be publicly available at https://videocomposer.github.io allowing researchers and practitioners to explore its capabilities further.