# Continual Pre-Training Mitigates Forgetting in Language and Vision

**Authors:** Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Davide Bacciu  
**arXiv:** [2205.09357](https://arxiv.org/abs/2205.09357)  
**Date:** 2022-05-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.09357v1/

## Key points

- The paper explores the use of pre-trained models in continual learning
- Pre-training is commonly used in machine learning research but rarely applied during continual learning
- Continual pre-training is formalized and investigated in both language and vision environments
- Models are continually pre-trained on incoming data before fine-tuning for different tasks
- Continually pre-trained models exhibit robustness against catastrophic forgetting
- Self-supervised pre-training is more effective than supervised protocols in retaining prior knowledge
- Code for the study is available at https://github.com/AndreaCossu/continual-pretraining-nlp-vision
- The authors are Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Bacciu
- The paper falls under computer science (cs.LG) and artificial intelligence (cs.AI)
- Incorporating continual pre-training can enhance model performance and mitigate catastrophic forgetting while updating with new data.

## Summary

The paper titled "Continual Pre-Training Mitigates Forgetting in Language and Vision" explores the use of pre-trained models in continual learning, where models are trained on a stream of non-stationary data. While pre-training is commonly used to initialize models in machine learning research, it is rarely applied during continual learning. The authors formalize and investigate the concept of continual pre-training in both language and vision environments. In this study, the authors continually pre-train models on incoming data before fine-tuning them for different downstream tasks. They demonstrate that continually pre-trained models exhibit robustness against catastrophic forgetting, which refers to the loss of previously learned knowledge when training on new data. The authors provide strong empirical evidence supporting the effectiveness of self-supervised pre-training compared to supervised protocols in retaining prior knowledge. To further support their findings, they have made code available at https://github.com/AndreaCossu/continual-pretraining-nlp-vision. The paper is authored by Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco and Davide Bacciu and falls under the categories of computer science (cs.LG) and artificial intelligence (cs.AI). Currently under review, this study highlights the potential benefits of incorporating continual pre-training into machine learning approaches for enhanced model performance and mitigation against catastrophic forgetting while updating models with new data without compromising prior knowledge retention.