# Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

**Authors:** Jonah Philion, Sanja Fidler  
**arXiv:** [2008.05711](https://arxiv.org/abs/2008.05711)  
**Date:** 2020-08-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2008.05711v1/

## Key points

- Authors: Jonah Philion and Sanja Fidler
- Objective: Extract semantic representations from various sensors and integrate them into a unified "bird's-eye-view" coordinate frame for motion planning
- Approach: Innovative end-to-end architecture generating bird's-eye-view representation using image data from multiple cameras
- Core concept: "Lifting" images into camera-specific frustums and "splatting" onto bird's-eye-view grid
- Model capabilities: Accurate representation of images, effective fusion of predictions from all cameras, resilience to calibration errors
- Performance: Superior on object segmentation and map segmentation tasks compared to existing baselines
- Motion planning: Dense representations enable interpretable end-to-end motion planning by projecting template trajectories onto bird's-eye-view cost map
- Validation: Benchmark tests against models using oracle depth from lidar technology demonstrate effectiveness and robustness
- Impact: Significant advancement in perception capabilities for autonomous vehicles, valuable insights for enhanced motion planning efficiency

## Summary

In their paper titled "Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D," authors Jonah Philion and Sanja Fidler address the crucial task of perception for autonomous vehicles. Their primary objective is to extract semantic representations from various sensors and integrate them into a unified "bird's-eye-view" coordinate frame for use in motion planning. To achieve this goal, the authors propose an innovative end-to-end architecture that directly generates a bird's-eye-view representation of a scene using image data captured by an arbitrary number of cameras. The core concept underlying their approach involves individually "lifting" each image into a frustum of features specific to each camera and then "splatting" all these frustums onto a rasterized bird's-eye-view grid. Through comprehensive training on the entire camera rig, the model demonstrates its ability not only to accurately represent images but also to effectively fuse predictions from all cameras into a cohesive scene representation while remaining resilient to calibration errors. Furthermore, the authors showcase the superior performance of their model on standard bird's-eye-view tasks such as object segmentation and map segmentation compared to existing baselines and prior research. In pursuit of learning dense representations for motion planning, they illustrate how the inferred representations enable interpretable end-to-end motion planning by projecting template trajectories onto a bird's-eye-view cost map generated by their network. Additionally, the authors conduct benchmark tests against models utilizing oracle depth from lidar technology, further validating the effectiveness and robustness of their proposed approach. Overall, their work presents a significant advancement in perception capabilities for autonomous vehicles and provides valuable insights into generating comprehensive scene representations for enhanced motion planning efficiency. More information about their project can be found on their project page with accompanying code at https://nv-tlabs.github.io/lift-splat-shoot.