# How to Teach Large Multimodal Models New Skills

**Authors:** Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem  
**arXiv:** [2510.08564](https://arxiv.org/abs/2510.08564)  
**Date:** 2025-10-09  
**Summary page:** https://www.summarizepaper.com/fr/arxiv-id/2510.08564v1/

## Abstract

How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. We observe that apparent "forgetting" on held-out tasks after narrow fine-tuning can partly recover at later stages. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that co-varies with forgetting. Guided by this picture, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers, and (ii) updating only the MLP Gate&Up while freezing the Down projection. Across models and tasks, these choices deliver strong target gains while largely preserving held-out performance. Code is available at https://github.com/jessemelpolio/LMM_CL