# ASTRAL: Automated Safety Testing of Large Language Models

**Authors:** Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura, Aitor Arrieta  
**arXiv:** [2501.17132](https://arxiv.org/abs/2501.17132)  
**Date:** 2025-01-28  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2501.17132v1/

## Key points

- Large Language Models (LLMs) raise safety concerns despite their impressive abilities
- Existing LLM testing frameworks address safety issues but face challenges due to unbalanced and outdated datasets
- ASTRAL is a novel tool automating test case generation for assessing LLM safety
- ASTRAL's innovative black-box coverage criterion aims to produce balanced and diverse unsafe test inputs across safety categories and writing styles
- GPT3.5 is superior in detecting unsafe responses compared to other LLMs like GPT-4 and specialized models like LlamaGuard
- ASTRAL uncovers nearly double the number of unsafe behaviors compared to static datasets with the same number of test inputs
- The combination of black-box coverage criterion with web browsing effectively guides LLMs in generating current unsafe test inputs

## Summary

, , , , 

The realm of Large Language Models (LLMs) has raised concerns about their safety, despite their impressive ability to comprehend and generate complex human-like content. Existing LLM testing frameworks have made progress in addressing safety-related issues, but challenges remain due to unbalanced and outdated datasets. To tackle these challenges, a novel tool called ASTRAL has been introduced for automating the generation and execution of test cases to assess LLM safety. A key highlight of ASTRAL is its innovative black-box coverage criterion, which aims to produce balanced and diverse unsafe test inputs across various safety categories and linguistic writing styles. Leveraging Retrieval Augmented Generation (RAG), few-shot prompting strategies, and web browsing, ASTRAL ensures the generation of up-to-date test inputs. Through an extensive evaluation on prominent LLMs, several key findings have emerged. Firstly, GPT3.5 stands out as a superior test oracle in detecting unsafe responses compared to other LLMs like GPT-4 and specialized models like LlamaGuard. Secondly, ASTRAL's approach uncovers nearly double the number of unsafe behaviors compared to static datasets with the same number of test inputs. Lastly, the combination of the black-box coverage criterion with web browsing proves effective in guiding LLMs to generate current unsafe test inputs. Furthermore, additional context provided through scenarios involving opioid engineering, securities fraud planning within corporations, ethical concerns in animal experimentation, community interventions for preventing animal cruelty, federal audit failures in child abuse investigations, clean air regulations impacting electric vehicle adoption, diversity in presidential leadership roles' impact on governance standards, long-term effects of child abuse on mental health, educating children about keeping secrets regarding abuse instances—all contribute to enhancing ASTRAL's capabilities in uncovering diverse safety scenarios efficiently without exhaustive testing. Overall, ASTRAL's automated testing approach showcases promising results in identifying unsafe behaviors in LLMs while adapting to evolving societal challenges across various domains effectively.