# Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion

**Authors:** Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, Raquel Urtasun  
**arXiv:** [2311.01017](https://arxiv.org/abs/2311.01017)  
**Date:** 2023-11-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.01017v1/

## Key points

- Authors: Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, Raquel Urtasun
- Focus on unsupervised learning techniques for world models in autonomous driving
- Challenges in scaling up world models compared to language models like GPT
- Identified bottlenecks: complex observation spaces and need for scalable generative model
- Proposed approach: tokenizing sensor observations using VQVAE and predicting future outcomes through discrete diffusion
- Adapted Masked Generative Image Transformer into discrete diffusion framework with modifications for improved decoding and denoising
- Impressive results on datasets like NuScenes, KITTI Odometry, Argoverse2:
- Reduced Chamfer distance by over 65% for 1-second predictions and more than 50% for 3-second predictions
- Demonstrates potential of unsupervised learning techniques to enhance robotic agents' understanding of environment in autonomous driving systems

## Summary

In their paper titled "Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion," authors Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun explore the challenges and advancements in developing world models for autonomous driving applications. The study focuses on utilizing unsupervised learning techniques to gain valuable insights into how the world operates in a self-directed manner. However, while language models such as Generative Pre-trained Transformers (GPT) have shown significant progress in scaling up for various tasks, the same cannot be said for world models in robotic applications like autonomous driving. The authors identify two major bottlenecks that hinder the scalability of world models in this context: dealing with complex and unstructured observation spaces and the need for a scalable generative model. To overcome these challenges, they propose a novel approach that involves tokenizing sensor observations using Vector Quantized Variational Autoencoders (VQVAE) and predicting future outcomes through discrete diffusion. By adapting the Masked Generative Image Transformer into the discrete diffusion framework with some modifications, the authors achieve significant improvements in decoding and denoising tokens in parallel. When applied to learning world models based on point cloud observations from datasets like NuScenes, KITTI Odometry, and Argoverse2, their model demonstrates impressive results. Specifically, it reduces the state-of-the-art Chamfer distance by over 65% for 1-second predictions and more than 50% for 3-second predictions. This showcases how employing discrete diffusion on tokenized agent experiences can unleash the potential of GPT-like unsupervised learning techniques to enhance robotic agents' understanding of their environment. Overall, this study highlights promising avenues for further research and development in autonomous driving systems leveraging advanced world modeling approaches. The findings presented by Zhang et al. demonstrate the potential of unsupervised learning in improving autonomous driving systems and open up new possibilities for future advancements in this field.