# End-to-end Autonomous Driving: Challenges and Frontiers

**Authors:** Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, Hongyang Li  
**arXiv:** [2306.16927](https://arxiv.org/abs/2306.16927)  
**Date:** 2023-06-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.16927v1/

## Key points

- Significant growth in the use of end-to-end algorithm frameworks in autonomous driving
- End-to-end systems offer joint feature optimization for perception and planning
- Factors contributing to this growth include large-scale datasets, closed-loop evaluation, and the need for effective performance in challenging scenarios
- A survey covering more than 250 papers provides a comprehensive analysis of this trend
- Critical challenges discussed include multi-modality, interpretability, causal confusion, robustness, and world models
- Causal confusion in imitation learning remains an open problem despite efforts to study it
- Dataset imbalance within long-tailed distributions is a severe issue in end-to-end autonomous driving
- Methods proposed to tackle dataset imbalance include over sampling, under sampling, data augmentation, and weighting based approaches
- Advancements in foundation models and visual pre-training techniques are discussed for incorporation into the end-to-end driving framework
- Modeling and utilizing uncertainty is emphasized for autonomous driving
- An active repository with links to relevant literature and open source projects is maintained for further research.

## Summary

In the field of autonomous driving, there has been a significant growth in the use of end-to-end algorithm frameworks which utilize raw sensor input to generate vehicle motion plans. This approach differs from traditional modular pipelines that focus on individual tasks such as detection and motion prediction. End-to-end systems offer the advantage of joint feature optimization for perception and planning. This growth can be attributed to factors such as the availability of large-scale datasets, closed-loop evaluation, and the increasing need for autonomous driving algorithms to perform effectively in challenging scenarios. To provide a comprehensive analysis of this trend, a survey was conducted covering more than 250 papers. The survey explores various aspects including motivation, roadmap, methodology, challenges and future trends in end-to-end autonomous driving. It delves into critical challenges such as multi-modality, interpretability, causal confusion, robustness and world models. One specific challenge discussed is causal confusion in imitation learning. Despite significant efforts to study this problem in recent years using modified environments for simplicity demonstrating performance improvements in state-of-the art settings remains an open problem. Another challenge highlighted is dataset imbalance within long tailed distributions. In the context of end to end autonomous driving this issue is particularly severe due to repetitive and uninteresting typical drives compared to rare but diverse safety critical scenarios. Various methods have been proposed to tackle dataset imbalance through data processing techniques such as over sampling under sampling data augmentation and weighting based approaches. The survey also discusses advancements in foundation models and visual pre training techniques and how these can be incorporated within the end to end driving framework. Additionally it emphasizes the importance of modeling and utilizing uncertainty for autonomous driving. To facilitate further research in this field an active repository containing up to date links to relevant literature and open source projects is maintained.