# Beyond Simulation: Benchmarking World Models for Planning and Causality in Autonomous Driving

**Authors:** Hunter Schofield, Mohammed Elmahgiubi, Kasra Rezaee, Jinjun Shan  
**arXiv:** [2508.01922](https://arxiv.org/abs/2508.01922)  
**Date:** 2025-08-03  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2508.01922v1/

## Key points

- World models are increasingly being used as learned traffic simulators for policy training.
- Recent research is shifting towards using world models for policy training instead of traditional traffic simulators.
- This study assesses the robustness of existing metrics for evaluating world models as traffic simulators and pseudo-environments for policy training.
- The researchers analyze the metametric used in the Waymo Open Sim-Agents Challenge (WOSAC) to compare world model predictions in various scenarios.
- The study extends the evaluation domain of WOSAC to include agents with a causal relationship with the ego vehicle, aiming to evaluate ego action-conditioned world models.
- New metrics are proposed to highlight the sensitivity of world models to uncontrollable objects and gauge their performance as pseudo-environments for policy training.
- Realistic simulation of causal agents influencing ego vehicle behavior is crucial for effective autonomous driving planning agent training.
- Penalizing planning agents for mistakes made by traffic simulation can lead to less well-defined behavior, potentially resulting in overly cautious driving strategies.
- This work introduces new metrics for assessing world models as data-driven traffic simulators, offering deeper insights into separate ego policy and traffic simulator performance compared to traditional evaluation methods like WOSAC's metametric.

## Summary

World models have gained popularity as learned traffic simulators. Recent research has focused on using them for policy training instead of traditional traffic simulators. This study delves into the robustness of existing metrics used to evaluate world models as traffic simulators and assesses their suitability for evaluating world models as pseudo-environments for policy training. The researchers specifically analyze the metametric utilized in the Waymo Open Sim-Agents Challenge (WOSAC) and compare world model predictions in standard scenarios where agents are either fully or partially controlled by the world model through partial replay. Additionally, this study extends the evaluation domain of WOSAC to include agents that have a causal relationship with the ego vehicle. This aims to assess the performance of ego action-conditioned world models. The evaluations uncover instances where top-ranking models excel under no perturbation but falter when the ego agent is compelled to replay its original trajectory. To address these challenges, new metrics are proposed to highlight the sensitivity of world models to uncontrollable objects and gauge their performance as pseudo-environments for policy training. It emphasizes the importance of evaluating causal agents, as discrepancies in traffic rollout performance are observed based on which domain of agents is being simulated. Realistic simulation of agents that influence the behavior of the ego vehicle throughout the simulation is crucial for effectively training an autonomous driving planning agent. Penalizing planning agents for mistakes made by the traffic simulation can lead to less well-defined behavior, potentially resulting in overly cautious or avoidant driving strategies. In conclusion, this work introduces new metrics for assessing world models as data-driven traffic simulators. It offers deeper insights into separate ego policy and traffic simulator performance compared to traditional evaluation methods like WOSAC's metametric. By exploring different evaluation domains and emphasizing realistic simulation of causal agents, this study contributes valuable perspectives on enhancing autonomous driving planning within world model simulations.