# Git Re-Basin: Merging Models modulo Permutation Symmetries

**Authors:** Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa  
**arXiv:** [2209.04836](https://arxiv.org/abs/2209.04836)  
**Date:** 2022-09-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.04836v1/

## Key points

- Deep learning success attributed to solving complex non-convex optimization problems easily
- Simple algorithms like stochastic gradient descent effective in fitting large neural networks
- Neural network loss landscapes typically contain a single basin, considering permutation symmetries of hidden units
- Introduction of three algorithms to permute units and align them with reference model
- Transformation results in functionally equivalent weights in approximately convex basin near reference model
- Experimental demonstration of single basin phenomenon across various model architectures and datasets
- Discovery of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10 and CIFAR-100 datasets
- Independently trained networks can have meaningful differences in learned features under certain scenarios
- Model width and training time affect mode connectivity across different models and datasets
- Shortcomings of single basin theory discussed, including counterexample to linear mode connectivity hypothesis
- Study provides insights into optimization landscape of neural networks and understanding convergence towards similar solutions

## Summary

The success of deep learning can be attributed to our ability to solve complex non-convex optimization problems with relative ease. Despite the NP-hard nature of non-convex optimization, simple algorithms like stochastic gradient descent have proven to be surprisingly effective in fitting large neural networks in practice. In this paper, the authors argue that neural network loss landscapes typically contain a single basin, taking into account all possible permutation symmetries of hidden units. To support their argument, the authors introduce three algorithms that permute the units of one model to align them with units of a reference model. This transformation results in a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. The authors experimentally demonstrate this single basin phenomenon across various model architectures and datasets. One notable finding is the discovery of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10 and CIFAR-100 datasets. This suggests that independently trained networks can exhibit meaningful differences in the features they learn under certain scenarios. Additionally, the authors identify intriguing phenomena related to model width and training time in relation to mode connectivity across different models and datasets. They also discuss shortcomings of the single basin theory, including a counterexample to the linear mode connectivity hypothesis. Overall, this study provides valuable insights into the optimization landscape of neural networks and sheds light on how different models can converge towards similar solutions despite their initial differences. The findings have implications for improving training algorithms and understanding the behavior of deep learning models more effectively.