# FACE-AUDITOR: Data Auditing in Facial Recognition Systems

**Authors:** Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Yang Zhang  
**arXiv:** [2304.02782](https://arxiv.org/abs/2304.02782)  
**Date:** 2023-04-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.02782v1/

## Key points

- Facial recognition systems are scalable and can work with few face images during model deployment
- These systems can be misused by entities who can build facial recognition models without people's awareness or consent
- Modifying raw face images before sharing them destroys semantic information and increases retroactivity difficulty
- The authors propose a complete toolkit called FACE-AUDITOR to audit facial recognition systems
- FACE-AUDITOR formulates the auditing process as a user-level membership inference problem
- The toolkit carefully chooses the probing set to query few-shot-based facial recognition models to determine if any of a user's face images were used in training the model
- The authors propose using similarity scores between original face images as reference information to improve auditing performance
- They consider three types of image-level similarity metrics: directly comparing pixel similarity, comparing structural similarity, or using deep neural networks for comparison
- Empirical experiments show that FACE-AUDITOR achieves up to 99% accuracy in multiple real-world face image datasets
- FACE-AUDITOR is robust in the presence of several perturbation mechanisms applied to training images or target models
- Cosine similarity achieves relatively better performance in most settings when learning similarity between images.

## Summary

Facial recognition systems have gained popularity due to their scalability and ability to work with a few face images during the model deployment phase. However, these systems can be misused by entities with moderate resources who can canvas the internet and build well-performed facial recognition models without people's awareness and consent. To prevent misuse of face images, one straightforward approach is to modify raw face images before sharing them, but this destroys semantic information and increases retroactivity difficulty. In this paper titled "FACE-AUDITOR: Data Auditing in Facial Recognition Systems," the authors formulate the auditing process as a user-level membership inference problem and propose a complete toolkit called FACE-AUDITOR. This toolkit carefully chooses the probing set to query few-shot-based facial recognition models and determines whether any of a user's face images are used in training the model. The authors further propose using similarity scores between original face images as reference information to improve auditing performance. To improve auditing performance further, they consider using image-level similarity between query image and support set as additional reference information referred to as reference auditing feature χr. The auditing feature χ is a concatenation of basic auditing feature and reference auditing feature (χ = χb||χr). They consider three types of image-level similarity metrics: directly comparing pixel similarity (MSE and CosSim), comparing structural similarity (SSIM), or using deep neural networks (LPIPS) for comparison. Empirical experiments show that FACE-AUDITOR achieves up to 99% accuracy in multiple real-world face image datasets. Additionally, FACE-AUDITOR is robust in the presence of several perturbation mechanisms applied to training images or target models. The authors also show that cosine similarity achieves relatively better performance in most settings when learning similarity between images. In conclusion, FACE-AUDITOR is an effective auditing tool that can determine whether facial recognition models use a user's face images without their consent. The proposed reference information and similarity metrics improve the auditing performance significantly. This paper will appear in the 32nd USENIX Security Symposium in August 2023, Anaheim, CA, USA.