# THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

**Authors:** Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li  
**arXiv:** [2206.00216](https://arxiv.org/abs/2206.00216)  
**Date:** 2022-06-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.00216v1/

## Key points

- The paper addresses privacy concerns in on-cloud deployment of pre-trained language models.
- Plain-text user data exposure poses significant risks to user privacy.
- Demand exists for privacy-preserving inference of transformer models in cloud services.
- THE-X introduces an approximation approach for transformers using homomorphic encryption (HE).
- THE-X enables privacy-preserving inference of pre-trained models with minimal performance degradation.
- Experimental results show that THE-X allows for transformer inference on encrypted data for various tasks while ensuring privacy preservation.
- Weight decay regularization impacts the attention overflow phenomenon and can improve approximation results.
- Larger weight decay may have varying effects on different tasks, potentially harming performance on some but benefiting others.
- Overall, THE-X offers a promising solution for enabling privacy-preserving inference of transformer models in cloud environments.

## Summary

The paper "THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption" addresses the growing privacy concerns associated with on-cloud deployment of pre-trained language models. The exposure of plain-text user data, such as search history, medical records, and bank account information, poses significant risks to user privacy. To mitigate these risks, there is a demand for privacy-preserving inference of transformer models in cloud services. One attractive solution is to perform computations only with ciphertext using homomorphic encryption (HE). However, enabling inference of pre-trained models on ciphertext data is challenging due to the complex computations involved in transformer blocks that are not currently supported by existing HE tools. In response to this challenge, the authors introduce THE-X, an approximation approach for transformers that enables privacy-preserving inference of pre-trained models developed by popular frameworks. THE-X proposes a workflow to handle complex computations in transformer networks, including non-polynomial functions like GELU, softmax, and LayerNorm. Experimental results demonstrate that THE-X allows for transformer inference on encrypted data for various downstream tasks with minimal performance degradation while ensuring theory-guaranteed privacy preservation. The study also investigates the impact of weight decay regularization on attention overflow phenomenon across different tasks. Without regularization, uncontrolled attention scores lead to poor performance; however, increasing weight decay helps converge attention scores and improve approximation results. The authors observe that larger weight decay may have varying effects on different tasks - potentially harming performance on NLI tasks but benefiting NER tasks and boosting the overall performance of their approximation method. Overall, THE-X offers a promising solution for enabling privacy-preserving inference of transformer models in cloud environments while maintaining high levels of data security and confidentiality.