# Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

**Authors:** Rui Li, Tobias Fischer, Mattia Segu, Marc Pollefeys, Luc Van Gool, Federico Tombari  
**arXiv:** [2404.03658](https://arxiv.org/abs/2404.03658)  
**Date:** 2024-04-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.03658v1/

## Key points

- The paper introduces a novel method called "Know Your Neighbors" (KYN) for recovering 3D scene geometry from a single view in computer vision.
- Traditional depth estimation methods are limited to inferring a 2.5D representation on the image plane, while recent approaches using radiance fields aim to reconstruct a full 3D representation.
- KYN leverages semantic and spatial context to accurately predict point density, overcoming limitations of existing methods in handling occluded regions.
- A key innovation of KYN is the integration of a vision-language modulation module that enriches point features with fine-grained semantic information.
- By incorporating language-guided spatial attention mechanisms, KYN aggregates point representations across the scene to generate per-point density predictions informed by 3D semantic context.
- Experimental results show that KYN significantly improves 3D shape recovery compared to traditional methods, achieving state-of-the-art performance in both scene and object reconstruction tasks on the KITTI-360 dataset.
- KYN also demonstrates enhanced zero-shot generalization capabilities compared to prior works in the field, showcasing its effectiveness in various scenarios.
- In conclusion, "Know Your Neighbors" offers an innovative solution by integrating semantic knowledge and spatial reasoning for single-view scene reconstruction, contributing valuable insights to advancing computer vision applications related to 3D scene understanding and reconstruction.

## Summary

The paper presents a novel method called "Know Your Neighbors" (KYN) for recovering 3D scene geometry from a single view in computer vision. Traditional depth estimation methods are limited to inferring a 2.5D representation on the image plane, but recent approaches utilizing radiance fields aim to reconstruct a full 3D representation. However, these methods often struggle with occluded regions due to the difficulty of inferring geometry without direct visual observation. To overcome these limitations, the authors propose KYN which leverages semantic and spatial context to accurately predict point density. The key innovation is the integration of a vision-language modulation module that enriches point features with fine-grained semantic information. By incorporating language-guided spatial attention mechanisms, KYN aggregates point representations across the scene to generate per-point density predictions informed by 3D semantic context. Experimental results demonstrate that KYN significantly improves 3D shape recovery compared to traditional methods that predict density for each 3D point independently. It achieves state-of-the-art performance in both scene and object reconstruction tasks on the KITTI-360 dataset and showcases enhanced zero-shot generalization capabilities compared to prior works in the field. In conclusion, "Know Your Neighbors" presents an innovative solution to the challenging problem of single-view scene reconstruction by effectively integrating semantic knowledge and spatial reasoning into the process. This research contributes valuable insights towards advancing computer vision applications related to 3D scene understanding and reconstruction.