# How to Teach Large Multimodal Models New Skills

**Authors:** Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem  
**arXiv:** [2510.08564](https://arxiv.org/abs/2510.08564)  
**Date:** 2025-10-09  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2510.08564v1/

## Key points

- Study focuses on teaching large multimodal models (LMMs) new skills without erasing prior abilities
- Sequential fine-tuning conducted on five target skills with monitoring of general ability on eight held-out benchmarks across three model families
- "Forgetting" observed on held-out tasks after narrow fine-tuning, but performance loss can partly recover at later stages
- Measurable shift in output token distribution identified through counting-bias probe correlated with forgetting
- Shift in performance driven by late MLP blocks rather than self-attention layers
- Two tuning recipes proposed: updating only self-attention projection layers or updating only MLP Gate&Up layers while freezing Down projection
- Strategies allow for effective learning while limiting drift across different model families
- Insights provided into learning and forgetting behavior of LMMs with practical recommendations to enhance model adaptation without sacrificing prior knowledge
- Acknowledgment of limitations such as resource constraints and need for further exploration with larger models and additional modalities like audio
- Foundation set for future research in areas such as alternative architectures, longer sequences, privacy leakage, safety considerations, and societal impacts

## Summary

This study focuses on teaching large multimodal models (LMMs) new skills without erasing their prior abilities. The researchers conducted sequential fine-tuning on five target skills and monitored general ability on eight held-out benchmarks across three model families. They observed that there was apparent "forgetting" on held-out tasks after narrow fine-tuning, but this loss in performance could partly recover at later stages. This behavior was linked to a measurable shift in the output token distribution, which was identified through a simple counting-bias probe that correlated with forgetting. Further analysis revealed that most of the shift in performance was driven by late MLP blocks rather than self-attention layers. Based on these findings, two simple and robust tuning recipes were proposed: updating only the self-attention projection layers or updating only the MLP Gate&Up layers while freezing the Down projection. These tuning strategies allowed for effective learning while limiting drift across different model families. Overall, this study provides insights into the learning and forgetting behavior of LMMs and offers practical recommendations to enhance model adaptation without sacrificing prior knowledge. The researchers hope that these findings will contribute to more stable and efficient continuous improvement of LMMs, ultimately reducing the environmental and financial costs associated with model adaptation. The study acknowledges limitations such as resource constraints and the need for further exploration with larger models and additional modalities like audio but sets a foundation for future research in areas such as alternative architectures, longer sequences, privacy leakage, safety considerations, and societal impacts. The work is supported by ONR award N00014-23-1-2383 and U.S. DARPA ECOLE Program No. #HR00112390060.