# DreamActor-M1: Holistic, Expressive and Robust Human Image Animation with Hybrid Guidance

**Authors:** Yuxuan Luo, Zhengkun Rong, Lizhen Wang, Longhao Zhang, Tianshu Hu, Yongming Zhu  
**arXiv:** [2504.01724](https://arxiv.org/abs/2504.01724)  
**Date:** 2025-04-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2504.01724v2/

## Key points

- Advancements in image-based human animation have improved realistic body and facial motion synthesis
- Existing methods face limitations in fine-grained holistic controllability, multi-scale adaptability, and long-term temporal coherence
- Introduction of diffusion transformer (DiT) based framework called DreamActor-M1 with hybrid guidance to address limitations
- Hybrid control signals integrate implicit facial representations, 3D head spheres, and 3D body skeletons for robust control over expressions and movements
- Progressive training strategy used for scale adaptation to handle various body poses and image scales
- Integration of motion patterns from sequential frames ensures long-term temporal coherence during complex movements
- Experimental results show outperformance of state-of-the-art works in delivering expressive animations with robust consistency
- Ablation studies highlight effectiveness of hybrid control signals in achieving high-quality human image animation
- Focus on multi-scale driven synthesis, fine-grained face and body control, and long-term temporal consistency for unseen areas to address challenges in image-based animation
- Innovative techniques such as hybrid control signals integration and progressive training strategies demonstrate promising results in generating lifelike human animations

## Summary

In recent years, advancements in image-based human animation have made significant strides in achieving realistic body and facial motion synthesis. However, there are still critical gaps in fine-grained holistic controllability, multi-scale adaptability, and long-term temporal coherence that limit the expressiveness and robustness of existing methods. To address these limitations, we introduce a diffusion transformer (DiT) based framework called DreamActor-M1 with hybrid guidance. Our approach incorporates hybrid control signals that integrate implicit facial representations, 3D head spheres, and 3D body skeletons to achieve robust control over facial expressions and body movements while maintaining expressive and identity-preserving animations. To handle various body poses and image scales ranging from portraits to full-body views, we employ a progressive training strategy using data with varying resolutions and scales for scale adaptation. Additionally, we integrate motion patterns from sequential frames with complementary visual references to ensure long-term temporal coherence for unseen regions during complex movements. Experimental results demonstrate that our method outperforms state-of-the-art works by delivering expressive results for portraits, upper-body poses, and full-body animations with robust long-term consistency. Quantitative comparisons on a collected dataset show improvements in metrics such as FID (Fréchet Inception Distance), SSIM (Structural Similarity Index Measure), PSNR (Peak Signal-to-Noise Ratio), LPIPS (Learned Perceptual Image Patch Similarity), and FVD (Fréchet Video Distance) when compared to portrait animation methods. Ablation studies further highlight the effectiveness of our hybrid control signals in achieving high-quality and realistic human image animation. Furthermore, recent developments in image-based animation have explored various directions but continue to face challenges in attaining photorealistic, expressive, and adaptable generation for practical applications. Our work aims to address these challenges by focusing on multi-scale driven synthesis, fine-grained face and body control, as well as long-term temporal consistency for unseen areas. By tackling these issues through innovative techniques such as hybrid control signals integration and progressive training strategies for scale adaptation, our framework demonstrates promising results in generating lifelike human animations for a wide range of scenarios including portrait reenactment to full-body dancing sequences. This comprehensive approach not only enhances the quality of generated animations but also paves the way for more versatile and adaptable image-based human animation systems.