# Humans in 4D: Reconstructing and Tracking Humans with Transformers

**Authors:** Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, Jitendra Malik  
**arXiv:** [2305.20091](https://arxiv.org/abs/2305.20091)  
**Date:** 2023-05-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.20091v1/

## Key points

- Authors present an innovative approach for reconstructing and tracking human bodies over time
- Utilizes a fully transformer-based network called HMR 2.0 for accurate 3D mesh recovery from single images
- Surpasses previous methods by effectively analyzing unusual poses that were previously challenging to reconstruct
- Leverages the capabilities of the transformer model within HMR 2.0
- Employs 3D reconstructions obtained from HMR 2.0 as input to a tracking system operating in 3D space for video analysis
- Enables handling scenarios involving multiple individuals and maintaining their identities during occlusion events
- Approach named 4DHumans achieves state-of-the-art results in tracking people from monocular video footage
- Demonstrates effectiveness of HMR 2.0 on action recognition tasks with significant improvements compared to previous approaches
- Provides access to code and models on project website for replication and further research

## Summary

In their paper titled "Humans in 4D: Reconstructing and Tracking Humans with Transformers," authors Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik present an innovative approach for reconstructing and tracking human bodies over time. Their method utilizes a fully transformer-based network called HMR 2.0 for accurate 3D mesh recovery from single images. The authors highlight that their approach surpasses previous methods by effectively analyzing unusual poses that were previously challenging to reconstruct. They achieve this by leveraging the capabilities of the transformer model within HMR 2.0. To analyze videos, they employ the 3D reconstructions obtained from HMR 2.0 as input to a tracking system operating in 3D space. This tracking system enables the researchers to handle scenarios involving multiple individuals and maintain their identities even during occlusion events. The complete approach, named 4DHumans, achieves state-of-the-art results in tracking people from monocular video footage. Additionally, the authors demonstrate the effectiveness of HMR 2.0 on action recognition tasks by achieving significant improvements compared to previous pose-based action recognition approaches. The paper provides access to their code and models on their project website (https://shubham-goel.github.io/4dhumans/), allowing other researchers to replicate and build upon their work. In summary, this paper introduces a novel transformer-based approach for recovering and tracking human body meshes in both images and videos.