# Continuous 3D Perception Model with Persistent State

**Authors:** Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, Angjoo Kanazawa  
**arXiv:** [2501.12387](https://arxiv.org/abs/2501.12387)  
**Date:** 2025-01-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.12387v1/

## Key points

- Introduction of CUT3R (Continuous Updating Transformer for 3D Reconstruction)
- Stateful recurrent model continuously updating state representation
- Generation of metric-scale pointmaps for each input image
- Ability to handle varying lengths of images (video streams, unordered photo collections)
- Competitive performance in various 3D/4D tasks
- Inference of new structures unobserved in input views through probing virtual views
- Simultaneous state-update and state-readout operations for each observation in an image stream

## Summary

In their paper titled "Continuous 3D Perception Model with Persistent State," Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa introduce a unified framework for solving a wide range of 3D tasks. Their approach involves a stateful recurrent model that continuously updates its state representation with each new observation in an online fashion. By processing a stream of images, this evolving state generates metric-scale pointmaps for each input, which can be accumulated into a coherent scene reconstruction that updates as new images arrive. Referred to as CUT3R (Continuous Updating Transformer for 3D Reconstruction), the model captures rich priors of real-world scenes and can predict accurate pointmaps from image observations while inferring unseen regions by probing virtual views. The authors highlight the simplicity and flexibility of their method, which can handle varying lengths of images such as video streams or unordered photo collections containing static and dynamic content. They evaluate CUT3R on various 3D/4D tasks and demonstrate competitive or state-of-the-art performance in each scenario. The model's ability to infer new structures unobserved in input views by probing the state with a raymap showcases its effectiveness in capturing generalized 3D scene priors. In conclusion, the authors propose an online  model with a continuously updating  that simultaneously performs state-update and state-readout operations for each observation in an image stream. The output includes camera parameters and pointmaps in the world frame, contributing to a dense reconstruction of the scene over time. Despite potential drift over long sequences, the method proves effective across various tasks and holds promise for future advancements in online  for .