# Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

**Authors:** James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, Hongxia Jin  
**arXiv:** [2304.06027](https://arxiv.org/abs/2304.06027)  
**Date:** 2023-04-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2304.06027v1/

## Key points

- Text-to-image generation is a growing research area that aims to synthesize high-quality images from textual descriptions
- Recent advances have shown progress in generating fine-grain details and customized concepts with few example images
- Continual customization of text-to-image models is a challenge when new concepts arrive sequentially, leading to catastrophic forgetting
- C-LoRA (Continually Self-Regularized Low-Rank Adaptation) efficiently adapts to new concepts while preserving knowledge of past concepts through continual self-regularized low-rank adaptation in cross attention layers of Stable Diffusion
- C-LoRA outperforms several baselines for Continual Diffusion and achieves state-of-the-art performance in rehearsal-free continual learning for image classification
- C-LoRA induces only marginal additional parameter costs and requires no storage of user data for replay
- Text-to-image generation has potential applications such as generating product images for ecommerce websites and creating personalized avatars for virtual reality environments
- C LoRA enables continuous customization without requiring retraining on past concepts after each new concept is added

## Summary

Text-to-image generation is a rapidly growing research area that aims to develop models capable of synthesizing high-quality images from textual descriptions. Recent advances in this area have shown significant progress in generating images with fine-grain details that can be customized to specific concepts, such as generating a portrait of one’s self or pet, while providing only a few example images to “instruct” the model. However, a practical question arises: What happens when these concepts arrive in a sequential manner? To address this challenge, we propose the setting of continual customization of text-to-image models, or simply Continual Diffusion. We show that current approaches suffer from catastrophic forgetting in this setting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high-quality images of past similar concepts degrades. To overcome this issue, we introduce C-LoRA (Continually Self-Regularized Low-Rank Adaptation), which efficiently adapts to new concepts while preserving the knowledge of past concepts through continual self-regularized low-rank adaptation in cross attention layers of Stable Diffusion. Our proposed method not only outperforms several baselines for Continual Diffusion but also achieves state-of-the-art performance in rehearsal-free continual learning for image classification. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. Text-to-image generation has the potential to enable a wide range of applications such as generating realistic product images for ecommerce websites and creating personalized avatars for virtual reality environments. Our work provides an efficient engagement with text-to image tools such as visual storytelling and content creation by enabling continuous customization without requiring retraining on past concepts after each new concept is added. The high achieving performance of C LoRA in two separate domains positions it as a compelling solution for various applications and has significant potential for practical impact.