# Scaling 4D Representations

**Authors:** João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, Goker Erdogan, Yana Hasson, Yi Yang, Klaus Greff, Guillaume Le Moing, Sjoerd van Steenkiste, Daniel Zoran, Drew A. Hudson, Pedro Vélez, Luisa Polanía  
**arXiv:** [2412.15212](https://arxiv.org/abs/2412.15212)  
**Date:** 2024-12-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.15212v1/

## Key points

- Authors focus on scaling in self-supervised learning from video data for non-semantic vision tasks
- Study explores tasks like camera pose estimation, point and object tracking, and depth estimation
- Scaling achieved by using large video datasets and masked auto-encoding (MAE) with transformer video models
- Performance improvements seen as model size increases from 20 million to 22 billion parameters
- Introduction of new collection of model checkpoints called 4DS ranging from 20 million to 22 billion parameters
- Significant performance improvements observed on spatial-temporal tasks by scaling MAE
- Comparison highlights benefits of scaling 4D representations over recent image and video models
- Challenges common belief about mediocre scaling properties of MAE through scaling up transformer models
- Contributions include re-evaluation of state-of-the-art scene representation models, introduction of three new MAE-VIT models within the 4DS family, and novel decoding scheme for efficient training of the largest model.

## Summary

The authors address the issue of scaling in self-supervised learning from video data by focusing on non-semantic vision tasks that are more spatial and temporal in nature. While prior work has primarily evaluated self-supervised learning on semantic-related tasks such as action classification and ImageNet classification, this study delves into tasks like camera pose estimation, point and object tracking, and depth estimation. The authors demonstrate that scaling is achievable by leveraging very large video datasets and utilizing masked auto-encoding (MAE) with transformer video models. They show consistent performance improvements on these 4D tasks as the model size increases from 20 million to a staggering 22 billion parameters. Through rigorous comparisons with recent image and video models, the benefits of scaling 4D representations are highlighted. Furthermore, the paper introduces a new collection of model checkpoints called 4DS, which includes models ranging from 20 million to 22 billion parameters. The authors emphasize that scaling MAE beyond what has been previously explored in literature brings about significant improvements in performance on these spatial-temporal tasks. The study also sheds light on the limitations of using language supervision alone compared to video self-supervision. By diving into MAE and scaling up transformer models from smaller sizes to the largest reported self-supervised video model thus far (22B parameters), the authors challenge the common belief in the community regarding mediocre scaling properties of MAE. Overall, the contributions of this work include a re-evaluation of state-of-the-art models for scene representation quality, the introduction of three new MAE-VIT models with varying parameter sizes within the 4DS family (2B, 4B, and 22B), as well as a novel decoding scheme for efficient training of the largest model. The paper structure covers related work, methodology details including baseline models and evaluation metrics, results showcasing performance improvements with increasing model sizes, before concluding with insights for future research directions.