Language-Guided Traffic Simulation via Scene-Level Diffusion

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cited by this paper Used in depth by this paper A method it uses Not relevant

   Guided Conditional Diffusion for Controllable Traffic Simulation

   Zi-Yuan Zhong, Davis Rempe, Dan-Fei Xu et al. · 2022 · IEEE International Conference on Robotics and Automation · cited 305 times · Not summarized yet

   Abstract

   Controllable and realistic traffic simulation is critical for developing and verifying autonomous vehicles. Typical heuristic-based traffic models offer flexible control to make vehicles follow specific trajectories and traffic rules. On the other hand, data-driven approaches generate realistic and human-like behaviors, improving transfer from simulated to real-world traffic. However, to the best of our knowledge, no traffic model offers both controllability and realism. In this work, we develop a conditional diffusion model for controllable traffic generation (CTG) that allows users to control desired properties of trajectories at test time (e.g., reach a goal or follow a speed limit) while…

2. Very close Not relevant

   Language Conditioned Traffic Generation

   Shuhan Tan, B. Ivanovic, Xinshuo Weng et al. · 2023 · Conference on Robot Learning · cited 126 times · Not summarized yet

   Abstract

   Simulation forms the backbone of modern self-driving development. Simulators help develop, test, and improve driving systems without putting humans, vehicles, or their environment at risk. However, simulators face a major challenge: They rely on realistic, scalable, yet interesting content. While recent advances in rendering and scene reconstruction make great strides in creating static scene assets, modeling their layout, dynamics, and behaviors remains challenging. In this work, we turn to language as a source of supervision for dynamic traffic scene generation. Our model, LCTGen, combines a large language model with a transformer-based decoder architecture that selects likely map location…

3. Very close Cites this paper Uses this paper in depth Not relevant

   Controllable Traffic Simulation through LLM-Guided Hierarchical Reasoning and Refinement

   Zhi-Yuan Liu, Le-Heng Li, Yuning Wang et al. · 2024 · IEEE/RJS International Conference on Intelligent RObots and Systems · cited 4 times · Not summarized yet

   Abstract

   Evaluating autonomous driving systems in complex and diverse traffic scenarios through controllable simulation is essential to ensure their safety and reliability. However, existing traffic simulation methods face challenges in their controllability. To address this, we propose a novel diffusion-based and LLM-enhanced traffic simulation framework. Our approach incorporates a high-level understanding module and a low-level refinement module, which systematically examines the hierarchical structure of traffic elements, guides LLMs to thoroughly analyze traffic scenario descriptions step by step, and refines the generation by self-reflection, enhancing their understanding of complex situations.…

4. Very close Cites this paper Not relevant

   LangTraj: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation

   Wei-Jer Chang, Wei Zhan, Masayoshi Tomizuka et al. · 2025 · IEEE International Conference on Computer Vision · cited 15 times · Not summarized yet

   Abstract

   Evaluating autonomous vehicles with controllability allows for scalable testing in counterfactual or structured settings, improving both efficiency and safety. We introduce LangTraj, a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenar-ios. By conditioning on natural language inputs, LANGTraj enables flexible and intuitive control over interactive behaviors, generating nuanced and realistic scenarios. Unlike prior approaches that rely on domain-specific guidance functions, LangTraj incorporates language conditioning during training for more intuitive traffic simulation control. In addition, we propose a novel closed-loop training stra…

5. Very close Not relevant

   Controllable Latent Diffusion for Traffic Simulation

   Yizhuo Xiao, M. Erden, Cheng Wang · 2025 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   The validation of autonomous driving systems benefits greatly from the ability to generate scenarios that are both realistic and precisely controllable. Conventional approaches, such as real-world test drives, are not only expensive but also lack the flexibility to capture targeted edge cases for thorough evaluation. To address these challenges, we propose a controllable latent diffusion that guides the training of diffusion models via reinforcement learning to automatically generate a diverse and controllable set of driving scenarios for virtual testing. Our approach removes the reliance on large-scale real-world data by generating complex scenarios whose properties can be finely tuned to c…

6. Very close Cites this paper Not relevant

   Safety-Critical Traffic Simulation with Guided Latent Diffusion Model

   Ming-Xing Peng, Ruo-Yu Yao, Xu-Sen Guo et al. · 2025 · 2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC) · cited 6 times · Not summarized yet

   Abstract

   Safety-critical traffic simulation plays a crucial role in evaluating autonomous driving systems under rare and challenging scenarios. However, existing approaches often generate unrealistic scenarios due to insufficient consideration of physical plausibility and suffer from low generation efficiency. To address these limitations, we propose a guided latent diffusion model (LDM) capable of generating physically realistic and adversarial safety-critical traffic scenarios. Specifically, our model employs a graph-based variational autoencoder (VAE) to learn a compact latent space that captures complex multi-agent interactions while improving computational efficiency. Within this latent space, t…

7. Very close Cites this paper Not relevant

   SceneControl: Diffusion for Controllable Traffic Scene Generation

   Jackie Lu, Kelvin Wong, Chris Zhang et al. · 2024 · IEEE International Conference on Robotics and Automation · cited 47 times · Not summarized yet

   Abstract

   We consider the task of traffic scene generation. A common approach in the self-driving industry is to use manual creation to generate scenes with specific characteristics and automatic generation to generate canonical scenes at scale. However, manual creation is not scalable, and automatic generation typically use rules-based algorithms that lack realism. In this paper, we propose SceneControl, a framework for controllable traffic scene generation. To capture the complexity of real traffic, SceneControl learns an expressive diffusion model from data. Then, using guided sampling, we can flexibly control the sampling process to generate scenes that exhibit desired characteristics. Our experim…

8. Very close Not relevant

   Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion

   Ethan Pronovost, Meghana Reddy Ganesina, Noureldin Hendy et al. · 2023 · Neural Information Processing Systems · cited 95 times · Not summarized yet

   Abstract

   Automated creation of synthetic traffic scenarios is a key part of validating the safety of autonomous vehicles (AVs). In this paper, we propose Scenario Diffusion, a novel diffusion-based architecture for generating traffic scenarios that enables controllable scenario generation. We combine latent diffusion, object detection and trajectory regression to generate distributions of synthetic agent poses, orientations and trajectories simultaneously. To provide additional control over the generated scenario, this distribution is conditioned on a map and sets of tokens describing the desired scenario. We show that our approach has sufficient expressive capacity to model diverse traffic patterns …

Prior work it builds on

1. Close A method it uses Not relevant

   TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors

   Simon Suo, S. Regalado, Sergio Casas et al. · 2021 · Computer Vision and Pattern Recognition · cited 346 times · Not summarized yet

   Abstract

   Simulation has the potential to massively scale evaluation of self-driving systems, enabling rapid development as well as safe deployment. Bridging the gap between simulation and the real world requires realistic multi-agent behaviors. Existing simulation environments rely on heuristic-based models that directly encode traffic rules, which cannot capture irregular maneuvers (e.g., nudging, U-turns) and complex interactions (e.g., yielding, merging). In contrast, we leverage real-world data to learn directly from human demonstration, and thus capture more naturalistic driving behaviors. To this end, we propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In parti…

2. Close A method it uses Not relevant

   Diffusion-LM Improves Controllable Text Generation

   Xiang Lisa Li, John Thickstun, I. Gulrajani et al. · 2022 · Neural Information Processing Systems · cited 1,661 times · Not summarized yet

   Abstract

   Controlling the behavior of language models (LMs) without re-training is a major open problem in natural language generation. While recent works have demonstrated successes on controlling simple sentence attributes (e.g., sentiment), there has been little progress on complex, fine-grained controls (e.g., syntactic structure). To address this challenge, we develop a new non-autoregressive language model based on continuous diffusions that we call Diffusion-LM. Building upon the recent successes of diffusion models in continuous domains, Diffusion-LM iteratively denoises a sequence of Gaussian vectors into word vectors, yielding a sequence of intermediate latent variables. The continuous, hier…

3. Related Used in depth by this paper A method it uses Not relevant

   Planning with Diffusion for Flexible Behavior Synthesis

   Michael Janner, Yilun Du, J. Tenenbaum et al. · 2022 · International Conference on Machine Learning · cited 1,533 times · Not summarized yet

   Abstract

   Model-based reinforcement learning methods often use learning only for the purpose of estimating an approximate dynamics model, offloading the rest of the decision-making work to classical trajectory optimizers. While conceptually simple, this combination has a number of empirical shortcomings, suggesting that learned models may not be well-suited to standard trajectory optimization. In this paper, we consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem, such that sampling from the model and planning with it become nearly identical. The core of our technical approach lies in a diffusion probabilistic model that plans b…

4. Close Not relevant

   BITS: Bi-level Imitation for Traffic Simulation

   Dan-Fei Xu, Yuxiao Chen, B. Ivanovic et al. · 2022 · IEEE International Conference on Robotics and Automation · cited 130 times · Not summarized yet

   Abstract

   Simulation is the key to scaling up validation and verification for robotic systems such as autonomous vehicles. Despite advances in high-fidelity physics and sensor simulation, a critical gap remains in simulating realistic behaviors of road users. This is because devising first principle models for human-like behaviors is generally infeasible. In this work, we take a data-driven approach to generate traffic behaviors from real-world driving logs. The method achieves high sample efficiency and behavior diversity by exploiting the bi-level hierarchy of high-level intent inference and low-level driving behavior imitation. The method also incorporates a planning module to obtain stable long-ho…

5. Related A method it uses Not relevant

   Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion

   Davis Rempe, Zhengyi Luo, Xue-Bin Peng et al. · 2023 · Computer Vision and Pattern Recognition · cited 196 times · Not summarized yet

   Abstract

   We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. We draw on recent advances in guided diffusion modeling to achieve test-time controllability of trajectories, which is normally only associated with rule-based systems. Our guided diffusion model allows users to constrain trajectories through target waypoints, speed, and specified social groups while accounting for the surrounding environment context. This trajectory diffusion model is integrated with a novel physics-based humanoid controller to form a closed-loop, full-body pedestrian animation system capable of placing large crowds in a simulate…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios

   Ming-Xing Peng, Yuting Xie, Xu-Sen Guo et al. · 2025 · Transportation Research Part C: Emerging Technologies · cited 23 times · Not summarized yet

   Abstract

   Ensuring the safety and robustness of autonomous driving systems necessitates a comprehensive evaluation in safety-critical scenarios. However, these safety-critical scenarios are rare and difficult to collect from real-world driving data, posing significant challenges to effectively assessing the performance of autonomous vehicles. Typical existing methods often suffer from limited controllability and lack user-friendliness, as extensive expert knowledge is essentially required. To address these challenges, we propose LD-Scene, a novel framework that integrates Large Language Models (LLMs) with Latent Diffusion Models (LDMs) for user-controllable adversarial scenario generation through natu…

2. Very close Not relevant

   Versatile Behavior Diffusion for Generalized Traffic Agent Simulation

   Zhi-Yu Huang, Zixu Zhang, Ameya Vaidya et al. · 2024 · IEEE transactions on intelligent transportation systems (Print) · cited 43 times · Not summarized yet

   Abstract

   Existing traffic simulation models often fall short in capturing the intricacies of real-world scenarios, particularly the interactive behaviors among multiple traffic participants, thereby limiting their utility in the evaluation and validation of autonomous driving systems. We introduce Versatile Behavior Diffusion (VBD), a novel traffic scenario generation framework based on diffusion generative models that synthesizes scene-consistent, realistic, and controllable multi-agent interactions. VBD achieves strong performance in closed-loop traffic simulation, generating scene-consistent agent behaviors that reflect complex agent interactions. A key capability of VBD is inference-time scenario…

3. Close Uses this paper in depth Not relevant

   CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning

   Luke Rowe, Roger Girgis, Anthony Gosselin et al. · 2024 · Conference on Robot Learning · cited 36 times · Not summarized yet

   Abstract

   Evaluating autonomous vehicle stacks (AVs) in simulation typically involves replaying driving logs from real-world recorded traffic. However, agents replayed from offline data are not reactive and hard to intuitively control. Existing approaches address these challenges by proposing methods that rely on heuristics or generative models of real-world data but these approaches either lack realism or necessitate costly iterative sampling procedures to control the generated behaviours. In this work, we take an alternative approach and propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning (RL) to efficiently generate reactive and controllable traffic agents. S…

4. Close Not relevant

   DragTraffic: Interactive and Controllable Traffic Scene Generation for Autonomous Driving

   Sheng Wang, Ge Sun, Fu-Long Ma et al. · 2024 · IEEE/RJS International Conference on Intelligent RObots and Systems · cited 10 times · Not summarized yet

   Abstract

   Evaluating and training autonomous driving systems require diverse and scalable corner cases. However, most existing scene generation methods lack controllability, accuracy, and versatility, resulting in unsatisfactory generation results. Inspired by DragGAN in image generation, we propose DragTraffic, a generalized, interactive, and controllable traffic scene generation framework based on conditional diffusion. DragTraffic enables non-experts to generate a variety of realistic driving scenarios for different types of traffic agents through an adaptive mixture expert architecture. We employ a regression model to provide a general initial solution and a refinement process based on the conditi…

5. Close Not relevant

   SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries

   Wei-Jer Chang, F. Pittaluga, Masayoshi Tomizuka et al. · 2023 · European Conference on Computer Vision · cited 67 times · Not summarized yet

   Abstract

   Evaluating the performance of autonomous vehicle planning algorithms necessitates simulating long-tail safety-critical traffic scenarios. However, traditional methods for generating such scenarios often fall short in terms of controllability and realism; they also neglect the dynamics of agent interactions. To address these limitations, we introduce SAFE-SIM, a novel diffusion-based controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations.…