# GARField: Group Anything with Radiance Fields

**Authors:** Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, Angjoo Kanazawa  
**arXiv:** [2401.09419](https://arxiv.org/abs/2401.09419)  
**Date:** 2024-01-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.09419v1/

## Key points

- Group Anything with Radiance Fields (GARField) is a method for decomposing 3D scenes into semantically meaningful groups
- GARField embraces group ambiguity through physical scale
- GARField optimizes a scale-conditioned 3D affinity feature field to allow for flexible and nuanced grouping decisions
- GARField uses 2D masks from Segment Anything (SAM) to generate initial groupings
- GARField fuses conflicting masks from different viewpoints using scale to ensure consistency and coherence in hierarchical grouping
- GARField can derive a hierarchy of possible groupings automatically or with user interaction
- GARField is effective in extracting groups at multiple levels, including clusters of objects, individual objects, and subparts
- GARField produces higher fidelity groups compared to SAM masks
- GARField's hierarchical grouping has potential applications in 3D asset extraction and dynamic scene understanding
- Quantitative evaluation shows that GARField consistently produces view-consistent groups and achieves high recall compared to ground truth human annotations

## Summary

, , , , 

Grouping is a challenging task in computer vision due to the ambiguity of how to decompose a scene into meaningful groups. The authors propose a method called Group Anything with Radiance Fields (GARField) that addresses this challenge by decomposing 3D scenes into a hierarchy of semantically meaningful groups. The key idea behind GARField is to embrace group ambiguity through physical scale. By optimizing a scale-conditioned 3D affinity feature field, a point in the world can belong to different groups of different sizes, allowing for more flexible and nuanced grouping decisions. To optimize this field, the authors use 2D masks provided by another method called Segment Anything (SAM), which generates initial groupings. To ensure consistency and coherence in the hierarchical grouping, GARField uses scale to fuse conflicting masks from different viewpoints. This ensures that the resulting groups are multi-view consistent and accurately represent the underlying scene structure. From the optimized affinity field, GARField can derive a hierarchy of possible groupings either automatically or with user interaction. The authors evaluate GARField on various real-world scenes and demonstrate its effectiveness in extracting groups at multiple levels, including clusters of objects, individual objects, and subparts. They compare GARField's results with the input SAM masks and find that GARField produces higher fidelity groups. Furthermore, GARField's hierarchical grouping has potential applications in 3D asset extraction and dynamic scene understanding. The authors provide visualizations of tree decompositions produced by their method, illustrating how objects gradually decompose into their constituent parts. In terms of quantitative evaluation, GARField is compared against annotated images using two metrics: view consistency and recall of hierarchical masks. The results show that GARField consistently produces view-consistent groups and achieves high recall compared to ground truth human annotations. Overall, GARField presents an innovative approach to addressing the ambiguity in grouping 3D scenes. Its ability to capture multi-view consistent groupings and produce high-quality hierarchical groupings has promising implications for various computer vision tasks.