# The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning

**Authors:** Borja Rodríguez-Gálvez, Arno Blaas, Pau Rodríguez, Adam Goliński, Xavier Suau, Jason Ramapuram, Dan Busbridge, Luca Zappella  
**arXiv:** [2307.10907](https://arxiv.org/abs/2307.10907)  
**Date:** 2023-07-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.10907v1/

## Key points

- The paper explores the mechanisms behind the success of multi-view self-supervised learning (MVSSL) and its relationship with mutual information (MI).
- The authors introduce a new lower bound on MI called entropy and reconstruction (ER), consisting of an entropy term and a reconstruction term.
- Various MVSSL methods are analyzed using this ER bound.
- Clustering-based methods like DeepCluster and SwAV maximize MI according to the ER bound.
- Distillation-based approaches like BYOL and DINO explicitly maximize the reconstruction term and implicitly encourage stable entropy.
- Empirical evidence supports this interpretation.
- The authors validate their findings by replacing objectives of common MVSSL methods with the ER bound, observing competitive performance while ensuring stability during training with smaller batch sizes or smaller exponential moving average coefficients.
- The paper includes acknowledgments for valuable feedback from reviewers, productive discussions with colleagues at Apple, and funding information for one of the authors.
- A GitHub repository link is provided for further reference.

## Summary

The paper explores the mechanisms behind the success of multi-view self-supervised learning (MVSSL) and aims to understand the relationship between MVSSL methods and mutual information (MI). The authors introduce a different lower bound on MI called entropy and reconstruction (ER), which consists of an entropy term and a reconstruction term. They analyze various MVSSL methods using this ER bound. The authors show that clustering-based methods like DeepCluster and SwAV maximize MI according to the ER bound. They also reinterpret distillation-based approaches such as BYOL and DINO, demonstrating that they explicitly maximize the reconstruction term and implicitly encourage a stable entropy. This interpretation is supported by empirical evidence. To validate their findings, the authors replace the objectives of common MVSSL methods with the ER bound. They observe competitive performance while ensuring stability during training with smaller batch sizes or smaller exponential moving average coefficients. In addition to expanding on these findings, the paper includes acknowledgments for valuable feedback from reviewers, productive discussions with colleagues at Apple, and funding information for one of the authors. The GitHub repository link is provided for further reference. Overall, this paper provides insights into the role of entropy and reconstruction in MVSSL methods, shedding light on their effectiveness in capturing meaningful representations while achieving competitive performance under various conditions.