# DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

**Authors:** Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth K. Y. Wong, Zhenguo Li, Hengshuang Zhao  
**arXiv:** [2310.01412](https://arxiv.org/abs/2310.01412)  
**Date:** 2023-10-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.01412v1/

## Key points

- Significant advancements in autonomous driving in the past decade in both academic research and industry applications
- Lack of interpretability in decision-making processes is a major challenge hindering widespread adoption and further development
- DriveGPT4 is an interpretable end-to-end autonomous driving system leveraging multimodal large language models (LLMs)
- DriveGPT4 can interpret vehicle actions, provide transparent reasoning, engage in dialogue with users, and predict low-level control signals
- Utilizes a custom visual instruction tuning dataset tailored for autonomous driving tasks to learn from visual cues and textual instructions simultaneously
- Demonstrates superior performance compared to traditional methods and video understanding LLMs across multiple evaluation tasks
- Capable of zero-shot learning, seamlessly generalizing to new scenarios without additional training data
- Sets a new standard for future developments in enhancing interpretability in autonomous driving systems

## Summary

In the past decade, significant advancements have been made in both academic research and industry applications for autonomous driving. However, a major challenge that continues to hinder its widespread adoption and further development is the lack of interpretability in decision-making processes. Previous attempts using small language models have not been successful due to limitations in flexibility, generalization capabilities, and robustness. Recently, there has been a growing interest in multimodal large language models (LLMs) within the research community. Building on this trend, a team of researchers led by Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth K. Y. Wong, Zhenguo Li and Hengshuang Zhao introduced DriveGPT4 - an interpretable end-to-end autonomous driving system that leverages LLMs. <br>
DriveGPT4 stands out for its ability to interpret vehicle actions and provide transparent reasoning behind its decisions. Additionally,<br>
it can engage in interactive dialogue with human users by answering diverse questions related to its driving behavior.<br>
The system also excels at predicting low-level control signals for vehicles in an end-to-end manner.<br>

The key strength of DriveGPT4 lies in its utilization of a custom visual instruction tuning dataset tailored specifically for autonomous driving tasks.<br>
This dataset enables the model to effectively learn from visual cues and textual instructions simultaneously.<br>
Notably,<br>
DriveGPT4 represents a pioneering effort in developing an interpretable end-to-end autonomous driving system.<br>

When compared against traditional methods as well as video understanding LLMs across multiple evaluation tasks,<br>
DriveGPT4 demonstrates superior performance both qualitatively and quantitatively.<br>

One particularly noteworthy feature of DriveGPT4 is its ability to seamlessly generalize to new scenarios without requiring additional training data - a capability known as zero-shot learning.<br>

For more information on DriveGPT4 and access to the project page, visit https://tonyxuqaq.github.io/projects/DriveGPT4/.<br>
With its innovative approach towards enhancing interpretability in autonomous driving systems,<br>
DriveGPT4 sets a new standard for future developments in this field.