# On the Discriminability of Self-Supervised Representation Learning

**Authors:** Zeen Song, Wenwen Qiang, Changwen Zheng, Fuchun Sun, Hui Xiong  
**arXiv:** [2407.13541](https://arxiv.org/abs/2407.13541)  
**Date:** 2024-07-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.13541v1/

## Key points

- Recent advancements in self-supervised learning (SSL) have shown success in improving downstream visual tasks
- There is a noticeable gap between SSL and supervised learning (SL), especially in complex downstream tasks
- One key issue is the crowding problem within features learned by SSL methods, leading to lack of distinct separation between classes and high intra-class variance
- SSL objectives do not effectively constrain relationships between different samples and their augmentations, resulting in poor performance in complex tasks
- A theoretical framework highlights that the performance disparity between SSL and SL arises from SSL methods' inability to capture aggregation of similar augmentations and separation of dissimilar augmentations
- The Dynamic Semantic Adjuster (DSA) approach has been proposed as a learnable regulator to address this challenge by focusing on aggregating and separating samples while maintaining robustness to outliers
- Extensive empirical evaluations on benchmark datasets show that DSA significantly enhances feature aggregation and separation, bridging the performance gap between SSL and SL
- Zeen Song, Wenwen Qiang, Changwen Zheng from University of Chinese Academy of Sciences in Beijing, China
- Fuchun Sun from Tsinghua University in Beijing, China
- Hui Xiong from Hong Kong University of Science and Technology in China
- Additional experiments were conducted to provide further insights into the issue through toy experiments related to feature learning techniques like SimCLR and BYO.

## Summary

In the field of self-supervised learning (SSL), recent advancements have shown significant success in improving downstream visual tasks. However, there remains a noticeable gap between SSL and supervised learning (SL), particularly in more complex downstream tasks. One key issue identified is the crowding problem within features learned by SSL methods, where features from different classes lack distinct separation and features within the same class exhibit high intra-class variance. In contrast, SL ensures clear class separation. Upon further analysis, it has been concluded that SSL objectives do not effectively constrain the relationships between different samples and their augmentations. This lack of constraint enforcement leads to poor performance in complex tasks. A theoretical framework has been developed to highlight that the performance disparity between SSL and SL primarily arises from SSL methods' inability to capture the aggregation of similar augmentations and the separation of dissimilar augmentations. To address this challenge, a novel approach called Dynamic Semantic Adjuster (DSA) has been proposed as a learnable regulator. DSA focuses on aggregating and separating samples in the feature space while maintaining robustness to outliers. Through extensive empirical evaluations on various benchmark datasets, it has been demonstrated that DSA significantly enhances feature aggregation and separation, ultimately bridging the performance gap between SSL and SL. The study was conducted by Zeen Song, Wenwen Qiang, Changwen Zheng from the University of Chinese Academy of Sciences in Beijing, China; Fuchun Sun from Tsinghua University in Beijing, China; and Hui Xiong from Hong Kong University of Science and Technology in China. The researchers have contributed equally to this work. Additional experiments were carried out to provide further insights into this issue through toy experiments illustrating specific aspects related to feature learning techniques like SimCLR and BYO.