# Evolutionary Optimization of Model Merging Recipes

**Authors:** Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, David Ha  
**arXiv:** [2403.13187](https://arxiv.org/abs/2403.13187)  
**Date:** 2024-03-19  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2403.13187v1/

## Key points

- Authors introduce evolutionary algorithms for automating creation of robust foundation models
- Traditional model merging in Large Language Model (LLM) development limited by human intuition and domain knowledge
- Evolutionary approach autonomously identifies effective combinations of diverse open-source models
- Approach operates in parameter space and data flow space, enabling optimization beyond individual model weights
- Allows for cross-domain merging, creating innovative models like Japanese LLM with Math reasoning capabilities
- Japanese Math LLM achieved state-of-the-art performance on established benchmarks despite not being explicitly trained for such tasks
- Authors demonstrate effectiveness by generating culturally-aware Japanese Visual Language Model (VLM) that outperforms previous models
- Research contributes new state-of-the-art models to open-source community and introduces novel paradigm for automated model composition

## Summary

In their paper titled "Evolutionary Optimization of Model Merging Recipes," authors Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha introduce a groundbreaking application of evolutionary algorithms to automate the creation of robust foundation models. The traditional approach to model merging in Large Language Model (LLM) development has shown promise for its cost-effectiveness but is limited by human intuition and domain knowledge. To address this limitation, the authors propose an evolutionary approach that autonomously identifies effective combinations of diverse open-source models, leveraging their collective intelligence without the need for extensive additional training data or computational resources. This innovative approach operates in both parameter space and data flow space, enabling optimization beyond just the individual model weights. Notably, it allows for cross-domain merging, resulting in the creation of models such as a Japanese LLM with Math reasoning capabilities. Surprisingly, the Japanese Math LLM developed through this method achieved state-of-the-art performance on various established Japanese LLM benchmarks, surpassing models with significantly more parameters despite not being explicitly trained for such tasks. Furthermore, the authors demonstrate the effectiveness of their approach by generating a culturally-aware Japanese Visual Language Model (VLM) that excels in describing Japan's culture-specific content, outperforming previous Japanese VLMs. This work not only contributes new state-of-the-art models to the open-source community but also introduces a novel paradigm for automated model composition. By paving the way for exploring alternative and efficient approaches to foundation model development, this research opens up exciting possibilities for advancing artificial intelligence technologies.