# CoDeF: Content Deformation Fields for Temporally Consistent Video Processing

**Authors:** Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, Yujun Shen  
**arXiv:** [2308.07926](https://arxiv.org/abs/2308.07926)  
**Date:** 2023-08-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.07926v1/

## Key points

- CoDeF (Content Deformation Fields) is a video representation method for achieving temporally consistent video processing.
- It consists of two main components: the canonical content field and the temporal deformation field.
- The canonical content field aggregates static contents in the entire video, while the temporal deformation field records transformations from the canonical image to each frame along the time axis.
- CoDeF supports lifting image algorithms for video processing by applying an algorithm to the canonical image and leveraging the temporal deformation field.
- It can achieve image-to-image translation to video-to-video translation and keypoint detection to keypoint tracking without requiring training data.
- Experimental results show that CoDeF outperforms existing video-to-video translation approaches in terms of cross-frame consistency and tracking non-rigid objects like water and smog.

## Summary

CoDeF (Content Deformation Fields) is a novel video representation method that aims to achieve temporally consistent video processing. It consists of two main components: the canonical content field and the temporal deformation field. The canonical content field aggregates the static contents present in the entire video, while the temporal deformation field records the transformations from the canonical image to each individual frame along the time axis. To reconstruct a target video, these two fields are jointly optimized through a carefully tailored rendering pipeline. The optimization process incorporates regularizations that encourage the canonical content field to inherit semantics such as object shape from the video. This design allows CoDeF to support lifting image algorithms for video processing. By applying an image algorithm to the canonical image and leveraging the temporal deformation field, one can effortlessly propagate outcomes to an entire video. One notable advantage of CoDeF is its ability to lift image-to-image translation to video-to-video translation and keypoint detection to keypoint tracking without requiring any training data. This is achieved by deploying algorithms on only one image within CoDeF's framework. Experimental results demonstrate that CoDeF outperforms existing video-to-video translation approaches in terms of cross-frame consistency in processed videos. Furthermore, it even succeeds in tracking non-rigid objects like water and smog. In conclusion, CoDeF presents a promising approach for achieving temporally consistent video processing by utilizing a combination of a canonical content field and a temporal deformation field. Its lifting strategy enables efficient application of image algorithms to videos, leading to superior results compared to existing methods. More information about CoDeF can be found on its project page at https://qiuyu96.github.io/CoDeF/.