# ViPlanner: Visual Semantic Imperative Learning for Local Navigation

**Authors:** Pascal Roth, Julian Nubert, Fan Yang, Mayank Mittal, Marco Hutter  
**arXiv:** [2310.00982](https://arxiv.org/abs/2310.00982)  
**Date:** 2023-10-02  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2310.00982v1/

## Key points

- Real-time path planning in outdoor environments is challenging due to varied terrain, diverse obstacles, and the need for quick decision-making.
- Traditional geometric navigation solutions struggle with interpreting different terrain types and identifying traversable occurrences like stairs.
- ViPlanner is a novel approach that integrates geometric and semantic information for local path planning.
- ViPlanner leverages Imperative Learning and a differentiable formulation of a semantic costmap to discern terrains' traversability levels and accurately detect obstacles.
- ViPlanner is trained exclusively in simulation, showcasing adaptability to real-world environments without requiring real-world training data.
- Experimental results demonstrate ViPlanner's robustness against noise, zero-shot sim-to-real transfer capability, and superior performance compared to purely geometric-based approaches with a 38.02% decrease in traversability cost.
- The code and models associated with ViPlanner are publicly available on GitHub at https://github.com/leggedrobotics/viplanner.
- Related works such as DD-PPO for point-goal navigation learning and ISLAM for imperative SLAM contribute to advancements in robotic navigation systems.

## Summary

, , , , 

In the realm of robotic systems, real-time path planning in outdoor environments poses a significant challenge due to the varied terrain traversability, diverse obstacles, and the need for quick decision-making. Traditional approaches have predominantly focused on geometric navigation solutions that excel in handling structured geometric obstacles but fall short when it comes to interpreting different terrain types and their affordances at a semantic level. These methods also struggle to identify traversable geometric occurrences like stairs. To address these limitations, a novel approach called ViPlanner has been introduced. This learned local path planning system generates plans based on both geometric and semantic information. ViPlanner leverages Imperative Learning, where network weights are optimized end-to-end based on the planning task objective. A key aspect of this optimization is the use of a differentiable formulation of a semantic costmap, enabling the planner to discern between various terrains' traversability levels and accurately detect obstacles. The semantic information is classified into 30 classes using an RGB colorspace, effectively encoding multiple levels of traversability. Notably, ViPlanner showcases adaptability to diverse real-world environments without necessitating real-world training; it is trained exclusively in simulation, facilitating highly scalable training data generation. Experimental results highlight ViPlanner's robustness against noise, its capability for zero-shot sim-to-real transfer, and its superior performance compared to purely geometric-based approaches with a 38.02% decrease in terms of traversability cost. The code and models associated with ViPlanner are publicly available on GitHub (https://github.com/leggedrobotics/viplanner). Additionally, related works such as DD-PPO for point-goal navigation learning and ISLAM for imperative SLAM are cited as contributing to advancements in robotic navigation systems. Furthermore, references to deep residual learning for image recognition and Carla—an open urban driving simulator—are made within the broader context of research in this field.