# DINO as a von Mises-Fisher mixture model

**Authors:** Hariprasath Govindarajan, Per Sidén, Jacob Roll, Fredrik Lindsten  
**arXiv:** [2405.10939](https://arxiv.org/abs/2405.10939)  
**Date:** 2024-05-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.10939v1/

## Key points

- Siamese networks are popular in self-supervised pre-training methods
- DINO method uses cross-entropy loss function with $K$-dimensional probability vectors and softmax operation on dot product of representations and learned prototypes
- Representations in DINO are $L^2$-normalized, leading to a von Mises-Fisher mixture model interpretation
- DINO-vMF introduces normalization constants for cluster assignment probability computation, demonstrating stability even with larger models like ViT-Base
- The von Mises-Fisher mixture model enhances image representations and outperforms standard DINO in downstream tasks
- iBOT-vMF also shows performance enhancements compared to iBOT, indicating the relevance of the modification for other methods based on the DINO framework

## Summary

In the field of self-supervised pre-training, methods utilizing Siamese networks have gained popularity. One such method is DINO, which employs a cross-entropy loss function to compare $K$-dimensional probability vectors derived from representations and learned prototypes through a softmax operation on their dot product. Notably, the representations in DINO are $L^2$-normalized. This led to an intriguing discovery that DINO and its variations like iBOT can be viewed as a mixture model comprising von Mises-Fisher components. This interpretation reveals that DINO assumes uniform precision across all components when the prototypes are also $L^2$-normalized. Building upon this insight, a novel approach called DINO-vMF is proposed. It introduces normalization constants during cluster assignment probability computation. Unlike its predecessor, DINO-vMF demonstrates stability even with larger models like ViT-Base featuring unnormalized prototypes. The flexibility offered by the von Mises-Fisher mixture model proves advantageous in enhancing image representations. Extensive evaluations showcase that the pre-trained DINO-vMF model consistently outperforms standard DINO across various downstream tasks. Similar performance enhancements are observed for iBOT-vMF compared to iBOT, underscoring the relevance of the proposed modification for other methods stemming from the DINO framework. Authored by Hariprasath Govindarajan, Per Sidén, Jacob Roll, and Fredrik Lindsten, the study titled "DINO as a von Mises-Fisher mixture model" has been accepted at ICLR 2023. This research sheds light on how leveraging mixture models can improve self-supervised learning techniques based on Siamese networks like DINO and its derivatives.