# 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera

**Authors:** Christen Millerdurai, Diogo Luvizon, Viktor Rudnev, André Jonas, Jiayi Wang, Christian Theobalt, Vladislav Golyanik  
**arXiv:** [2312.14157](https://arxiv.org/abs/2312.14157)  
**Date:** 2023-12-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.14157v1/

## Key points

- Challenging problem of 3D hand tracking from a monocular video
- Factors to consider: hand interactions, occlusions, left-right hand ambiguity, and fast motion
- Existing methods rely on RGB inputs with limitations under low-light conditions and motion blur susceptibility
- Event cameras capture local brightness changes instead of full image frames and are not affected by these issues
- Significant differences in data modalities between event data and image-based techniques
- Proposed framework for 3D tracking of two fast-moving and interacting hands using a single monocular event camera
- Semi-supervised feature-wise attention mechanism to tackle left-right hand ambiguity in event data
- Integration of intersection loss to address collisions between hands during interactions
- Release of synthetic large-scale dataset (Ev2Hands-S) and real benchmark dataset (Ev2Hands-R) with ground truth 3D annotations
- Experimental results show superior 3D reconstruction accuracy compared to existing methods
- Generalizes well to real data even under severe light conditions
- Pioneering framework for 3D pose estimation of two interacting hands from monocular event camera data
- Addresses challenges such as hand interactions and occlusions while leveraging advantages offered by event cameras

## Summary

This paper addresses the challenging problem of 3D hand tracking from a monocular video, considering factors such as hand interactions, occlusions, left-right hand ambiguity, and fast motion. Existing methods for this task typically rely on RGB inputs, which have limitations under low-light conditions and are susceptible to motion blur. In contrast, event cameras capture local brightness changes instead of full image frames and are not affected by these issues. However, existing image-based techniques cannot be directly applied to event data due to significant differences in the data modalities. To overcome these challenges, the authors propose a novel framework for 3D tracking of two fast-moving and interacting hands using a single monocular event camera. The proposed approach tackles the left-right hand ambiguity by introducing a semi-supervised feature-wise attention mechanism. This mechanism helps distinguish between the left and right hands in the event data. Additionally, an intersection loss is integrated into the framework to address collisions between the hands during interactions. To facilitate further research in this domain, the authors release two valuable resources: a synthetic large-scale dataset called Ev2Hands-S, which consists of two interacting hands; and a real benchmark dataset called Ev2Hands-R that includes real event streams with ground truth 3D annotations. Experimental results demonstrate that the proposed approach outperforms existing methods in terms of 3D reconstruction accuracy. Moreover, it generalizes well to real data even under severe light conditions. In conclusion, this paper presents a pioneering framework for 3D pose estimation of two interacting hands from monocular event camera data. The proposed approach effectively addresses challenges such as hand interactions and occlusions while leveraging the advantages offered by event cameras. The released datasets contribute significantly to advancing research in this field.