# Guidelines for evaluation of complex multi agent test scenarios

**Authors:** Ana Isabel Garcia Guerra, Teng Sung Shiuan  
**arXiv:** [2405.10526](https://arxiv.org/abs/2405.10526)  
**Date:** 2024-05-17  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2405.10526v1/

## Key points

- Comprehensive guideline created by CETRAN to support testing of autonomous vehicles (AVs)
- Focus on evaluating complex multi-agent test scenarios in a structured manner
- Dissecting challenges faced by AVs in perception, situation awareness, and decision-making capabilities
- Guidelines consist of elements for assessing scenario complexity and evaluating AV behavior
- Tool to challenge risk management abilities of AVs in scenario-based testing or real-world traffic situations
- Evaluation of five real events on Singapore roads using Resembler webtool data
- Design of four simulation scenarios by CETRAN team to illustrate potential challenges faced by automated driving systems (ADS)
- Study provides insights into understanding and managing complexity for AVs
- Offers practical guidance for assessing AV performance under varying conditions

## Summary

In their study titled "Guidelines for evaluation of complex multi agent test scenarios," authors Ana Isabel Garcia Guerra and Teng Sung Shiuan present a comprehensive guideline created by CETRAN to support the testing of autonomous vehicles (AVs). The guideline aims to evaluate complex multi-agent test scenarios in a structured manner, focusing on the difficulties AVs may encounter in Singapore traffic. By dissecting the challenges faced by AVs in terms of perception, situation awareness, and decision-making capabilities, the study seeks to understand the sources of complexity arising from traffic hazards. The guidelines developed through this research consist of a list of elements that can be used as selection criteria to assess the complexity of scenarios and support the evaluation of AV behavior. This framework serves as a tool to challenge the risk management abilities of autonomous vehicles in scenario-based testing approaches or real-world traffic situations encountered during road trials. The report utilizes these guidelines to evaluate the complexity of five real events occurring on Singapore roads using data from Resembler webtool, which catalogues real human accidents/incidents. Additionally, four scenarios were designed by the CETRAN team for simulation purposes, applying the complexity elements outlined in the guidelines to illustrate potential challenges faced by automated driving systems (ADS) in such scenarios. Overall, this study provides valuable insights into understanding and managing complexity for AVs, offering practical guidance for assessing their performance under varying conditions. The findings contribute to enhancing safety measures and optimizing autonomous vehicle technology for future deployment on public roads.