# Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks

**Authors:** Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, David Scott Krueger  
**arXiv:** [2311.12786](https://arxiv.org/abs/2311.12786)  
**Date:** 2023-11-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.12786v1/

## Key points

- Fine-tuning large pre-trained models is a prevalent strategy in machine learning for crafting task-specific and versatile systems.
- There is a gap in understanding how fine-tuning impacts the fundamental capabilities acquired during pretraining.
- Researchers use synthetic, controlled environments and interpretability tools to analyze alterations in model abilities during fine-tuning.
- Fine-tuning typically adds a minimal transformation called a 'wrapper' on top of existing capabilities rather than introducing entirely new ones.
- Hidden capabilities can be efficiently revived with further fine-tuning on relevant tasks.
- Fine-tuning on unrelated tasks may unintentionally remove a model's protective 'safety wrapper'.
- Research extends to real-world datasets like TinyStories to fortify findings and understand the effects of fine-tuning on model capabilities.

## Summary

In the realm of machine learning, fine-tuning large pre-trained models has emerged as a prevalent strategy for crafting task-specific and versatile systems. This approach places particular emphasis on ensuring the safety of deployed models. However, despite its widespread adoption, there exists a notable gap in understanding how fine-tuning impacts the fundamental capabilities acquired by a model during its pretraining phase. The pivotal question at hand is whether fine-tuning introduces entirely new capabilities or simply adjusts existing ones. To delve into this inquiry, researchers have embarked on empirical investigations within synthetic, controlled environments. Here they can leverage mechanistic interpretability tools such as network pruning and probing to dissect the alterations in a model's underlying abilities. Through an extensive analysis conducted in these controlled settings, it has been revealed that fine-tuning seldom results in substantial changes to the core model capabilities. Instead, what typically occurs is the acquisition of a minimal transformation termed as a 'wrapper' atop the foundational capabilities of the model. This 'wrapper' essentially creates an illusion of modified capabilities while preserving the essence of the original learnings. Furthermore, when subjected to further fine-tuning on tasks where these latent capabilities are pertinent, there is a swift and efficient 'revival' of these hidden abilities within just a few gradient steps. This phenomenon underscores an intriguing aspect wherein practitioners may inadvertently strip away a model's protective 'safety wrapper' by fine-tuning it on seemingly unrelated downstream tasks. To fortify their findings in more practical scenarios, researchers have extended their analysis to language models trained on real-world datasets like TinyStories. The study titled "Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks" authored by Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger provides valuable insights into how fine-tuning influences model capabilities and emphasizes the importance of understanding these dynamics for developing robust and safe machine learning systems.