# VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

**Authors:** Adrien Bardes, Jean Ponce, Yann LeCun  
**arXiv:** [2105.04906](https://arxiv.org/abs/2105.04906)  
**Date:** 2021-05-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2105.04906v1/

## Key points

- Self-supervised learning for image representation has seen a surge in methods that maximize agreement between embedding vectors from different views of the same image.
- The collapse problem, where the encoder outputs constant vectors, is a common issue with such methods.
- Researchers have introduced a new method called Variance-Invariance-Covariance Regularization (VICReg) to address this problem.
- VICReg incorporates a regularization term on the variance of embeddings along each dimension individually and combines it with a decorrelation mechanism based on redundancy reduction and covariance regularization.
- VICReg achieves results on par with state-of-the-art methods on several downstream tasks and can be incorporated into other methods to stabilize training and lead to performance improvements.
- VICReg provides an explicit solution to avoiding collapse problems without relying on implicit biases in learning architecture that lack clear justification or interpretation.
- Overall, VICReg represents an important contribution to self-supervised learning for image representation by providing a more robust and interpretable approach to addressing collapse problems.

## Summary

The field of self-supervised learning for image representation has recently seen a surge in methods that aim to maximize the agreement between embedding vectors from different views of the same image. However, a common issue with such methods is the trivial solution obtained when the encoder outputs constant vectors, known as the collapse problem. To address this problem, researchers Adrien Bardes, Jean Ponce, and Yann LeCun have introduced a new method called Variance-Invariance-Covariance Regularization (VICReg). VICReg explicitly avoids the collapse problem by incorporating a simple regularization term on the variance of embeddings along each dimension individually. The method also combines this variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization. The authors demonstrate that VICReg achieves results on par with state-of-the-art methods on several downstream tasks and can be incorporated into other methods to stabilize training and lead to performance improvements. This approach provides an explicit solution to avoiding collapse problems without relying on implicit biases in learning architecture that lack clear justification or interpretation. Overall, VICReg represents an important contribution to self-supervised learning for image representation by providing a more robust and interpretable approach to addressing collapse problems. By explicitly avoiding these issues through its regularization terms and decorrelation mechanisms, VICReg offers an effective way to improve accuracy while maintaining interpretability in self-supervised learning for image representation.