# Vision-Only Robot Navigation in a Neural Radiance World

**Authors:** Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, Mac Schwager  
**arXiv:** [2110.00168](https://arxiv.org/abs/2110.00168)  
**Date:** 2021-10-01  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2110.00168v2/

## Key points

- Neural Radiance Fields (NeRFs) are a powerful approach for representing complex 3D scenes
- NeRFs use neural networks to model volumetric density and RGB values for realistic image generation
- This study proposes an algorithm for vision-only robot navigation using pre-trained NeRF representations
- A trajectory optimization algorithm is introduced to navigate the robot through unoccupied space in the NeRF, avoiding collisions
- An optimization-based filtering method estimates the robot's pose and velocities using only an onboard RGB camera
- The trajectory planner and pose filter are combined in an online replanning loop for continuous adaptation based on real-time perception
- Extensive simulations validate the approach in various scenarios, including quadrotor navigation through a jungle gym and ground robot navigation through narrow gaps in a church environment
- Videos showcasing simulated robot navigation can be accessed at [link to project's website]
- The study highlights the potential of this vision-based navigation pipeline for accurate localization, trajectory planning, and collision avoidance within a NeRF representation of a 3D environment.

## Summary

, , , , 


Neural Radiance Fields (NeRFs) have gained significant attention as a powerful approach for representing complex 3D scenes. NeRFs utilize neural networks to model continuous volumetric density and RGB values, enabling the generation of realistic images from novel camera viewpoints through ray tracing. In this study, we present an algorithm that leverages NeRFs for vision-only robot navigation in a 3D environment. Our proposed algorithm assumes that the NeRF representation of the scene has been pre-trained offline. The objective is to navigate the robot through unoccupied space within the NeRF to reach a desired pose or goal. To achieve this, we introduce a trajectory optimization algorithm that takes into account high-density regions in the NeRF to avoid collisions. This optimization algorithm is based on a discrete time version of differential flatness, which allows us to constrain the robot's full pose and control inputs. In addition, we propose an optimization-based filtering method to estimate the six degrees of freedom (6DoF) pose and velocities of the robot within the NeRF using only an onboard RGB camera. This filtering method enables accurate localization and tracking of the robot's position and orientation in real-time. To create a complete vision-based robot navigation pipeline, we combine the trajectory planner with the pose filter in an online replanning loop. This integration allows for continuous adaptation and adjustment of the robot's trajectory based on real-time perception from the RGB camera. We validate our approach through extensive simulations involving different scenarios, such as a quadrotor robot navigating through a jungle gym environment, exploring the interior of a church, and maneuvering around Stonehenge. Remarkably, all these navigation tasks are accomplished solely using visual information captured by an RGB camera mounted on-board. Furthermore, we demonstrate our algorithm's versatility by showcasing its effectiveness with an omnidirectional ground robot navigating through narrow gaps within a church environment. The robot successfully reorients itself to fit through the constrained spaces, highlighting the robustness and adaptability of our vision-based navigation approach. To provide a visual demonstration of our work, we have included videos showcasing the simulated robot navigation in various environments. These videos can be accessed at [link to the project's website]. In summary, our study presents a novel algorithm for  in a NeRF representation of a 3D environment. By leveraging  and an onboard RGB camera, our approach enables accurate localization, trajectory planning, and collision avoidance within the NeRF. The demonstrated results highlight the potential of this vision-based navigation pipeline for .