# MotionCLIP: Exposing Human Motion Generation to CLIP Space

**Authors:** Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, Daniel Cohen-Or  
**arXiv:** [2203.08063](https://arxiv.org/abs/2203.08063)  
**Date:** 2022-03-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.08063v1/

## Key points

- MotionCLIP is a 3D human motion auto-encoder with a disentangled and well-behaved latent embedding.
- The model aligns with the Contrastive Language-Image Pre-training (CLIP) model's latent space, infusing the manifold with CLIP's rich semantic knowledge.
- The transformer-based motion auto-encoder reconstructs motion while being aligned to its text label's position in CLIP-space.
- MotionCLIP leverages CLIP's visual understanding and aligns motion to rendered frames in a self-supervised manner for unprecedented text-to-motion abilities.
- The introduced latent space can be leveraged for motion interpolation, editing, and recognition.
- Limitations include struggling to understand directions or capturing certain styles accurately.
- MotionCLIP opens up several novel research opportunities such as generating signature motions of cultural figures and phrases.

## Summary

MotionCLIP is a novel 3D human motion auto-encoder that features a disentangled and well-behaved latent embedding which supports highly semantic textual descriptions. The model's unique power lies in its alignment with the Contrastive Language-Image Pre-training (CLIP) model's latent space. This alignment infuses the manifold with CLIP's rich semantic knowledge, enabling continuity by placing semantically similar motions close to one another and disentanglement inherited from the CLIP-space structure. The transformer-based motion auto-encoder is trained to reconstruct motion while being aligned to its text label's position in CLIP-space. Additionally, MotionCLIP leverages CLIP's visual understanding and injects an even stronger signal through aligning motion to rendered frames in a self-supervised manner. This enables unprecedented text-to-motion abilities such as out-of domain actions, disentangled editing, abstract language specification and semantic interpolation between two motions. For instance, when prompted with "couch," MotionCLIP generates a sitting down motion due to lingual similarity; similarly when prompted with "Spiderman," it produces a web swinging like solution that was not seen during training. The introduced latent space can also be leveraged for motion interpolation, editing and recognition. However, MotionCLIP has limitations such as struggling to understand directions or capturing certain styles accurately. Nonetheless it opens up several novel research opportunities such as generating signature motions of cultural figures and phrases. Overall, MotionCLIP represents a significant advancement in human motion generation by exposing it to CLIP space and leveraging its rich semantic knowledge for unprecedented text-to-motion abilities.