# Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis

**Authors:** Rachel S. Y. Teo, Tan M. Nguyen  
**arXiv:** [2406.13762](https://arxiv.org/abs/2406.13762)  
**Date:** 2024-06-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.13762v1/

## Key points

- Authors Rachel S. Y. Teo and Tan M. Nguyen focus on the success of transformers in sequence modeling tasks by examining self-attention mechanisms
- They introduce a novel approach using kernel principal component analysis (kernel PCA) to derive self-attention, projecting query vectors onto principal component axes within a feature space
- The authors formulate an exact formula for the value matrix in self-attention, capturing eigenvectors of the Gram matrix of key vectors
- Teo and Nguyen propose Attention with Robust Principal Components (RPC-Attention), a robust attention mechanism designed to withstand data contamination
- Empirical evaluations on tasks like ImageNet-1K object classification, WikiText-103 language modeling, and ADE20K image segmentation show advantages of RPC-Attention over traditional softmax attention methods
- RPC-Attention implemented in a Segmenter model for ADE20K image segmentation demonstrates superior performance on clean and corrupted data sets compared to baseline approaches
- Evaluation of RPC-Attention on WikiText-103 language modeling task shows improvements in validation and test perplexity metrics

## Summary

In their work titled "Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis," authors Rachel S. Y. Teo and Tan M. Nguyen explore the success of transformers in sequence modeling tasks by focusing on the critical role of self-attention mechanisms. They introduce a novel approach to deriving self-attention from kernel principal component analysis (kernel PCA), demonstrating how self-attention projects query vectors onto principal component axes within a feature space. The authors also formulate an exact formula for the value matrix in self-attention, showcasing its ability to capture eigenvectors of the Gram matrix of key vectors. Building upon this foundation, Teo and Nguyen propose Attention with Robust Principal Components (RPC-Attention), a robust attention mechanism designed to withstand data contamination. Through empirical evaluations on tasks such as ImageNet-1K object classification, WikiText-103 language modeling, and ADE20K image segmentation, they showcase the advantages of RPC-Attention over traditional softmax attention methods. Further expanding their study, the authors implement RPC-Attention in a Segmenter model for ADE20K image segmentation, demonstrating superior performance compared to baseline approaches on both clean and corrupted data sets. Additionally, they evaluate RPC-Attention on the WikiText-103 language modeling task by replacing standard transformer language models with RPC-enhanced versions in select layers. The results show improvements in validation and test perplexity metrics. Overall, Teo and Nguyen's research sheds light on the underlying structure of self-attention mechanisms through kernel PCA analysis and introduces a robust attention framework that shows promise across various applications in deep learning tasks. Their findings contribute valuable insights into enhancing transformer models for improved performance and resilience against data anomalies.