Results of the summarizing process for the arXiv paper: 2304.02782v1

Facial recognition systems have gained popularity due to their scalability and ability to work with a few face images during the model deployment phase. However, these systems can be misused by entities with moderate resources who can canvas the internet and build well-performed facial recognition models without people's awareness and consent. To prevent misuse of face images, one straightforward approach is to modify raw face images before sharing them, but this destroys semantic information and increases retroactivity difficulty. In this paper titled "FACE-AUDITOR: Data Auditing in Facial Recognition Systems," the authors formulate the auditing process as a user-level membership inference problem and propose a complete toolkit called FACE-AUDITOR. This toolkit carefully chooses the probing set to query few-shot-based facial recognition models and determines whether any of a user's face images are used in training the model. The authors further propose using similarity scores between original face images as reference information to improve auditing performance. To improve auditing performance further, they consider using image-level similarity between query image and support set as additional reference information referred to as reference auditing feature χr. The auditing feature χ is a concatenation of basic auditing feature and reference auditing feature (χ = χb||χr). They consider three types of image-level similarity metrics: directly comparing pixel similarity (MSE and CosSim), comparing structural similarity (SSIM), or using deep neural networks (LPIPS) for comparison. Empirical experiments show that FACE-AUDITOR achieves up to 99% accuracy in multiple real-world face image datasets. Additionally, FACE-AUDITOR is robust in the presence of several perturbation mechanisms applied to training images or target models. The authors also show that cosine similarity achieves relatively better performance in most settings when learning similarity between images. In conclusion, FACE-AUDITOR is an effective auditing tool that can determine whether facial recognition models use a user's face images without their consent. The proposed reference information and similarity metrics improve the auditing performance significantly. This paper will appear in the 32nd USENIX Security Symposium in August 2023, Anaheim, CA, USA.

• - Facial recognition systems are scalable and can work with few face images during model deployment
• - These systems can be misused by entities who can build facial recognition models without people's awareness or consent
• - Modifying raw face images before sharing them destroys semantic information and increases retroactivity difficulty
• - The authors propose a complete toolkit called FACE-AUDITOR to audit facial recognition systems
• - FACE-AUDITOR formulates the auditing process as a user-level membership inference problem
• - The toolkit carefully chooses the probing set to query few-shot-based facial recognition models to determine if any of a user's face images were used in training the model
• - The authors propose using similarity scores between original face images as reference information to improve auditing performance
• - They consider three types of image-level similarity metrics: directly comparing pixel similarity, comparing structural similarity, or using deep neural networks for comparison
• - Empirical experiments show that FACE-AUDITOR achieves up to 99% accuracy in multiple real-world face image datasets
• - FACE-AUDITOR is robust in the presence of several perturbation mechanisms applied to training images or target models
• - Cosine similarity achieves relatively better performance in most settings when learning similarity between images.

Facial recognition systems can recognize faces and work with few pictures. Sometimes, people can use these systems without permission. Changing pictures before sharing them makes it harder to use facial recognition. A toolkit called FACE-AUDITOR helps check if facial recognition is being used fairly. It compares pictures to see if they were used to train the system. The toolkit works well even when pictures are changed or distorted.

Facial Recognition Systems: A Closer Look at FACE-AUDITOR

In recent years, facial recognition systems have gained immense popularity due to their scalability and ability to work with a few face images during the model deployment phase. However, these systems can be misused by entities with moderate resources who can canvas the internet and build well-performed facial recognition models without people's awareness and consent. To prevent misuse of face images, one straightforward approach is to modify raw face images before sharing them, but this destroys semantic information and increases retroactivity difficulty. This is why researchers from the 32nd USENIX Security Symposium in August 2023, Anaheim, CA, USA have proposed a toolkit called FACE-AUDITOR that carefully chooses a probing set to query few-shot based facial recognition models and determine whether any of a user's face images are used in training the model. In this article we will take an in depth look at how FACE-AUDITOR works and its potential applications.

Formulating Auditing Process as User Level Membership Inference Problem

The authors formulate the auditing process as a user level membership inference problem which means that they use data from multiple users (or members) to infer whether or not certain data points belong to each individual user. This type of inference helps ensure that no single user’s data is used without their knowledge or consent. The authors propose using similarity scores between original face images as reference information to improve auditing performance further by considering image level similarity between query image and support set as additional reference information referred to as reference auditing feature χr. The auditing feature χ is then created by concatenating basic auditing feature with reference auditing feature (χ = χb||χr).

Types of Image Level Similarity Metrics Used

The authors consider three types of image level similarity metrics for comparison: directly comparing pixel similarity (MSE & CosSim), comparing structural similarity (SSIM), or using deep neural networks (LPIPS). Empirical experiments show that FACE-AUDITOR achieves up 99% accuracy in multiple real world datasets when using cosine similarity which suggests it may be more effective than other methods for learning similarities between images. Additionally, FACE-AUDITOR has proven robust even when perturbation mechanisms are applied either on target models or training images.

Conclusion

FACE-AUDITOR is an effective tool for determining whether facial recognition models use a user’s face image without their consent while also providing high levels of accuracy across different datasets regardless of perturbations applied on target models or training images. The proposed reference information combined with various types of image level similarities improves overall performance significantly making it an invaluable asset for anyone looking into preventing misuse of personal data through facial recognition systems