# NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

**Authors:** Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, Yu-Gang Jiang  
**arXiv:** [2305.14836](https://arxiv.org/abs/2305.14836)  
**Date:** 2023-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14836v1/

## Key points

- Introduction of a new task in visual question answering (VQA) for autonomous driving scenarios
- Goal is to answer natural language questions based on street-view clues
- Challenges include multi-modal data (images and point clouds), multi-frame data, and moving foreground/static background elements
- Existing VQA benchmarks do not adequately address these complexities
- Proposal of NuScenes-QA as the first benchmark for VQA in autonomous driving scenarios
- Benchmark includes 34K visual scenes and 460K question-answer pairs
- Creation of benchmark involved leveraging existing 3D detection annotations, generating scene graphs, and designing question templates
- Development of baselines using advanced 3D detection and VQA techniques
- Extensive experiments highlight challenges and provide insights for future research directions
- Access to codes and datasets related to NuScenes-QA provided through GitHub repository
- Contribution to advancing research in VQA for autonomous driving scenarios by addressing unique complexities

## Summary

The paper introduces a new task in the field of visual question answering (VQA) specifically designed for autonomous driving scenarios. The goal is to answer natural language questions based on street-view clues. This task poses several challenges compared to traditional VQA tasks. Firstly, the raw visual data in this context are multi-modal, consisting of images and point clouds captured by cameras and LiDAR sensors respectively. Secondly, the data are multi-frame due to continuous real-time acquisition. Lastly, outdoor scenes exhibit both moving foreground objects and static background elements. Existing VQA benchmarks fail to adequately address these complexities, which motivated the authors to propose NuScenes-QA as the first benchmark for VQA in autonomous driving scenarios. The benchmark includes 34K visual scenes and 460K question-answer pairs. To create this benchmark, the authors leverage existing 3D detection annotations to generate scene graphs and manually design question templates. Subsequently, programmatically generated question-answer pairs are based on these templates. Comprehensive statistics demonstrate that NuScenes-QA is a balanced large-scale benchmark with diverse question formats. Building upon this benchmark, the authors develop a series of baselines that employ advanced 3D detection and VQA techniques. Extensive experiments highlight the challenges posed by this new task while providing insights into potential solutions for future research directions in this domain. The paper concludes by providing access to codes and datasets related to NuScenes-QA through a GitHub repository (https://github.com/qiantianwen/NuScenes-QA). Overall, this work contributes significantly to advancing research in VQA for autonomous driving scenarios by addressing the unique complexities associated with such environments.