# Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

**Authors:** Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt  
**arXiv:** [2203.05482](https://arxiv.org/abs/2203.05482)  
**Date:** 2022-03-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.05482v1/

## Key points

- Authors introduce a novel approach for maximizing model accuracy in fine-tuning large pre-trained models
- They propose averaging the weights of multiple fine-tuned models with different hyperparameter configurations to create "model soups"
- Model soups improve accuracy and robustness without additional inference or memory costs
- The study focuses on models like CLIP, ALIGN, and ViT-G trained on JFT
- The model soup outperforms the best model in a hyperparameter sweep on ImageNet, achieving a new state-of-the-art top-1 accuracy of 90.94%
- The approach extends beyond image classification to natural language processing tasks, enhancing out-of-distribution and zero-shot performance
- An analytical connection is established between weight averaging and logit ensembling techniques to loss function flatness and prediction confidence levels
- This innovative approach enhances model performance by leveraging ensemble techniques through weight averaging without compromising efficiency or computational resources

## Summary

In their paper "Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time," authors Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith and Ludwig Schmidt introduce a novel approach for maximizing model accuracy in the context of fine-tuning large pre-trained models. Traditionally, this process involves training multiple models with different hyperparameters and selecting the best-performing one on a validation set. However,<DateTime> propose an alternative method where they average the weights of multiple models that have been fine-tuned with various hyperparameter configurations. This technique of creating "model soups" has shown promising results in improving both accuracy and robustness without incurring additional inference or memory costs. The study focuses specifically on fine-tuning large pre-trained models such as CLIP, ALIGN and a ViT-G model trained on JFT. Remarkably,<DateTime> soup recipe outperforms the best model in a hyperparameter sweep on ImageNet,<DateTime> resulting ViT-G model achieving a new state-of-the-art top-1 accuracy of 90.94% on ImageNet. Furthermore,<DateTime> demonstrate that this model soup approach extends beyond image classification tasks to natural language processing tasks.<DateTime> not only enhances out-of-distribution performance but also improves zero-shot performance on new downstream tasks. Moreover,<DateTime> establish an analytical connection between weight-averaging and logit-ensembling techniques to the flatness of loss functions and confidence levels of predictions.<DateTime>, this innovative approach presents a compelling strategy for enhancing model performance in various machine learning applications by leveraging ensemble techniques through weight averaging without compromising efficiency or computational resources.