# Polarized Self-Attention: Towards High-quality Pixel-wise Regression

**Authors:** Huajun Liu, Fuqiang Liu, Xinyi Fan, Dong Huang  
**arXiv:** [2107.00782](https://arxiv.org/abs/2107.00782)  
**Date:** 2021-07-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2107.00782v2/

## Key points

- The paper addresses challenges of pixel-wise regression in fine-grained computer vision tasks
- Attention mechanisms have become popular for boosting long-range dependencies, but element-specific attention is complex and noise-sensitive to learn
- The authors present the Polarized Self-Attention (PSA) block that incorporates two critical designs towards high-quality pixel-wise regression: polarized filtering and enhancement
- The PSA block appears to have exhausted the representation capacity within its channel only and spatial only branches
- Experimental results show that PSA boosts standard baselines by $2 - 4$ points and state of the art models by $1 - 2$ points on 2D pose estimation and semantic segmentation benchmarks.
- The proposed method achieves state of the art performance on benchmark datasets for 2D pose estimation and semantic segmentation tasks.

## Summary

The paper "Polarized Self-Attention: Towards High-quality Pixel-wise Regression" by Huajun Liu, Fuqiang Liu, Xinyi Fan, and Dong Huang addresses the challenges of pixel-wise regression in fine-grained computer vision tasks. These tasks involve estimating keypoint heatmaps and segmentation masks which require modeling long-range dependencies on high resolution inputs/outputs to estimate highly nonlinear pixel-wise semantics. While attention mechanisms in Deep Convolutional Neural Networks (DCNNs) have become popular for boosting long range dependencies, element specific attention such as Nonlocal blocks is complex and noise sensitive to learn. Most simplified attention hybrids try to reach the best compromise among multiple types of tasks. To address these challenges, the authors present the Polarized Self-Attention (PSA) block that incorporates two critical designs towards high quality pixel wise regression. The first design is polarized filtering which keeps high internal resolution in both channel and spatial attention computation while completely collapsing input tensors along their counterpart dimensions. The second design is enhancement which composes non linearity that directly fits the output distribution of typical fine grained regression such as the 2D Gaussian distribution (keypoint heatmaps) or the 2D Binormial distribution (binary segmentation masks). The PSA block appears to have exhausted the representation capacity within its channel only and spatial only branches such that there are only marginal metric differences between its sequential and parallel layouts. Experimental results show that PSA boosts standard baselines by $2 - 4$ points and state of the art models by $1 - 2$ points on 2D pose estimation and semantic segmentation benchmarks. In conclusion, this paper presents a novel approach to address the challenges of pixel wise regression in fine grained computer vision tasks using Polarized Self Attention blocks. The proposed method achieves state of the art performance on benchmark datasets for 2D pose estimation and semantic segmentation tasks.