# Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models

**Authors:** Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiangyong Zhu, Bo Zheng, Kaifu Zhang  
**arXiv:** [2412.15265](https://arxiv.org/abs/2412.15265)  
**Date:** 2024-12-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2412.15265v2/

## Key points

- Concerns surrounding safety in Large Language Models (LLMs) are increasingly prominent, especially in domains like law, policy, and ethics.
- The Chinese SafetyQA benchmark was introduced to assess LLMs' factuality abilities in answering short questions related to safety.
- Analysis of the Chinese Safety Domain dataset showed that o1-preview performed the best overall, while the gpt-4o-mini model had the lowest performance.
- GPT models excelled in Physical & Mental Health topics but struggled in Illegal & Regulatory Compliance topics compared to specialized Chinese models like Qwen-series and Doubao.
- Chinese models performed poorly on Safety Theoretical Knowledge topics, indicating a lack of understanding in areas such as network safety and information safety.
- Related works have focused on evaluating LLM factuality through benchmarks like SimpleQA and Chinese SimpleQA, as well as methods like self-reflection and RAG.
- The introduction of Chinese SafetyQA is a significant advancement for assessing LLM factuality specifically within the Chinese context, filling a crucial gap in existing safety benchmarks.

## Summary

In the rapidly evolving landscape of Large Language Models (LLMs), concerns surrounding their safety have become increasingly prominent. The accuracy, comprehensiveness, and clarity of LLMs' understanding of safety knowledge, particularly in domains like law, policy, and ethics, are crucial for ensuring their safe deployment and compliance within specific regions. To address these challenges and assess the factuality abilities of LLMs in answering short questions, the Chinese SafetyQA benchmark was introduced. This benchmark is characterized by being Chinese, diverse, high-quality, static, easy to evaluate, safety-related, and harmless. An analysis was conducted on the results of different subtopics within the Chinese Safety Domain dataset. It was observed that o1-preview performed the best across all categories while the gpt-4o-mini model exhibited the lowest performance. GPT models showed better proficiency in Physical & Mental Health topics due to more training on international ESG issues. However, non-Chinese models struggled in Illegal & Regulatory Compliance topics compared to Chinese models like Qwen-series and Doubao which displayed better performance due to specialized training efforts on Chinese legal knowledge. Furthermore,a comparison revealed that all Chinese models performed poorly on Safety Theoretical Knowledge topics indicating a lack of understanding in areas such as network safety and information safety. Related works have focused on evaluating LLM factuality and simple QA with benchmarks like SimpleQA and Chinese SimpleQA emphasizing ease of evaluation. Efforts have also been made to enhance LLM factuality through methods like self-reflection and RAG. In conclusion,the introduction of Chinese SafetyQA marks a significant advancement in assessing the factuality abilities of LLMs specifically within the Chinese context.This benchmark fills a crucial gap in existing safety benchmarks by focusing on compliance and legality evaluations for regions like China. Overall,this work contributes to enhancing our understanding of LLM capabilities in addressing safety-related challenges effectively.