# Differentially Private Neural Network Training under Hidden State Assumption

**Authors:** Ding Chen, Chen Liu  
**arXiv:** [2407.08233](https://arxiv.org/abs/2407.08233)  
**Date:** 2024-07-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.08233v1/

## Key points

- Deep neural networks have made significant advancements in face recognition and natural language processing.
- Concerns about privacy vulnerabilities have arisen due to reliance on large amounts of training data, leading to potential privacy breaches.
- Differential privacy has emerged as a crucial tool for quantitatively guaranteeing the privacy of machine learning models.
- One common approach is introducing calibrated noise during each update step in the training process to balance utility and minimize privacy loss.
- Existing methods for preserving differential privacy may be limited in scenarios with non-convex optimization problems and algorithms like proximal gradient descent.
- A novel approach called differentially private stochastic block coordinate descent (DP-SBCD) integrates Lipschitz neural networks and breaks down the training process into sub-problems corresponding to specific layers.
- DP-SBCD extends the analysis of differential privacy under the hidden state assumption to encompass non-convex optimization problems and algorithms utilizing proximal gradient descent.
- DP-SBCD introduces a strategy by leveraging calibrated noise sampled from adaptive distributions, enhancing the trade-off between utility and privacy compared to existing methods.

## Summary

In recent years, deep neural networks have made significant advancements in various fields such as face recognition and natural language processing. However, concerns about privacy vulnerabilities associated with these models have arisen due to their reliance on large amounts of training data. It has been shown that deep neural networks can memorize training data, potentially leading to privacy breaches. To address this issue, the concept of differential privacy has emerged as a crucial tool for quantitatively guaranteeing the privacy of machine learning models. One common approach to ensuring differential privacy in machine learning is by introducing calibrated noise during each update step in the training process. This creates a delicate balance between preserving utility and minimizing privacy loss. However, striking this balance can be challenging as excessive noise can degrade model performance while insufficient noise may compromise privacy. Existing methods for preserving differential privacy often make assumptions that limit their applicability in scenarios where non-convex optimization problems and algorithms like proximal gradient descent are used. To overcome these limitations, a novel approach called differentially private stochastic block coordinate descent (DP-SBCD) has been proposed. This methodology integrates Lipschitz neural networks and breaks down the neural network training process into sub-problems corresponding to specific layers. By doing so, DP-SBCD extends the analysis of differential privacy under the hidden state assumption to encompass non-convex optimization problems and algorithms utilizing proximal gradient descent. Moreover, DP-SBCD introduces a unique strategy by leveraging calibrated noise sampled from adaptive distributions. This innovative approach enhances the trade-off between utility and privacy compared to existing methods. Through incorporating adaptive noise sampling techniques, DP-SBCD aims to improve empirical outcomes while maintaining robust guarantees of differential privacy. Overall, this refined summary highlights the challenges posed by privacy vulnerabilities in deep neural networks and introduces an advanced methodology that offers provable guarantees of differential privacy under the hidden state assumption while optimizing trade-offs between utility and confidentiality.