# Understanding Catastrophic Forgetting in Language Models via Implicit Inference

**Authors:** Suhas Kotha, Jacob Mitchell Springer, Aditi Raghunathan  
**arXiv:** [2309.10105](https://arxiv.org/abs/2309.10105)  
**Date:** 2023-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.10105v1/

## Key points

- Fine-tuning is crucial in adapting pretrained models to specific tasks
- Lack of systematic understanding on the effects of fine-tuning, especially for tasks outside the narrow distribution used for fine-tuning
- Introduction of a synthetic setup to explore impact of fine-tuning on LLMs by pretraining transformers on diverse weight vectors
- Fine-tuning on limited datasets can lead to suboptimal performance on certain tasks, emphasizing the need for deeper understanding of trade-offs
- Proposal of Conjugate Prompting as a method to counteract negative effects of fine-tuning and recover pretrained model capabilities
- Application of Conjugate Prompting to real-world LLMs successfully restores some pretraining capabilities and addresses concerns related to harmful content generation in chatbots like ChatGPT
- Importance of developing comprehensive understanding of fine-tuning effects to enhance model robustness and adaptability across diverse tasks and datasets

## Summary

In the development of large language models (LLMs), fine-tuning plays a crucial role in adapting pretrained models to specific tasks of interest. However, there is a lack of systematic understanding regarding the effects of fine-tuning, especially on tasks that fall outside the narrow distribution used for fine-tuning. To address this issue, researchers have introduced a synthetic setup to explore the impact of fine-tuning on LLMs. By pretraining transformers on a diverse set of weight vectors and evaluating their performance on specific weight vectors, they aim to mimic real-world scenarios where uncurated pretraining data may not align with tasks of special interest. The study reveals that fine-tuning on limited datasets can lead to suboptimal performance on certain tasks, highlighting the need for a deeper understanding of the trade-offs involved in the fine-tuning process. The researchers propose Conjugate Prompting as a method to counteract the negative effects of fine-tuning and recover pretrained model capabilities. By artificially making tasks appear farther from the fine-tuning distribution while maintaining the same level of complexity, they demonstrate that it is possible to restore some pretraining capabilities in LLMs. Moreover, by applying Conjugate Prompting to real-world LLMs and leveraging language translation techniques to shift task inference away from English-centric distributions, they successfully recover lost in-context learning abilities and address concerns related to harmful content generation in chatbots like ChatGPT. Overall, this study sheds light on how fine-tuning impacts LLM performance and offers insights into mitigating catastrophic forgetting through innovative prompting strategies. The findings underscore the importance of developing a comprehensive understanding of fine-tuning effects to enhance model robustness and adaptability across diverse tasks and datasets.