# Deep Model Fusion: A Survey

**Authors:** Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, Li Shen  
**arXiv:** [2309.15698](https://arxiv.org/abs/2309.15698)  
**Date:** 2023-09-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.15698v1/

## Key points

- Deep model fusion combines parameters or predictions of multiple deep learning models into a single one
- The approach aims to enhance model performance by leveraging the strengths of different models
- Challenges in applying deep model fusion to large-scale models include high computational cost, interference between heterogeneous models, and a high-dimensional parameter space
- Methods for deep model fusion include mode connectivity, alignment, weight average, and ensemble learning
- Applications of deep model fusion include federated learning (FL), distillation, and large language models (LLMs)
- The survey identifies bottlenecks and breakthroughs in deep model fusion and provides directions for future research
- Novel strategies should be designed from innovative aggregation patterns, better initial conditions, diverse ensemble frameworks, etc.
- Untapped potential exists in exploring the loss landscape and uncovering relationships between network components
- Better adaptive methods are needed for heterogeneous models and complex real scenarios like FL and transfer learning
- Practical effects should be considered to promote the development and application of deep model fusion technologies

## Summary

Deep model fusion is a rapidly growing technique that combines the parameters or predictions of multiple deep learning models into a single one. This approach aims to enhance model performance by leveraging the strengths of different models to compensate for biases and errors. However, applying deep model fusion to large-scale models presents challenges such as high computational cost, interference between heterogeneous models, and a high-dimensional parameter space. To address these challenges, researchers have proposed various methods for deep model fusion. One method is mode connectivity which focuses on connecting solutions in weight space to obtain better initialization for model fusion. Another approach is alignment which matches units between neural networks to create optimal conditions for fusion. Weight average is a classical method that averages the weights of multiple models to achieve more accurate results closer to the optimal solution. Ensemble learning combines the outputs of diverse models and serves as a foundational technique for improving accuracy and robustness. In addition to discussing existing deep model fusion techniques, this survey also highlights their applications and engineering prospects in areas such as federated learning (FL), distillation, large language models (LLMs), etc. The survey identifies bottlenecks and breakthroughs in deep model fusion and provides valuable directions for future research. Moving forward, it is suggested that novel strategies should be designed from innovative aggregation patterns, better initial conditions, diverse ensemble frameworks, and other perspectives. There is still untapped potential in exploring the information in the loss landscape and uncovering potential relationships between network components. Furthermore, better adaptive methods are needed for heterogeneous models and complex real scenarios like FL, large-scale models, transfer learning etc. Practical effects should also be considered to promote the development and application of deep model fusion technologies. Overall,this comprehensive survey provides insights into different deep model fusion methods and their practical applications. It serves as a valuable resource for developers looking to enhance the performance of deep model fusion technologies and identifies promising directions for future research and development.