# Heterogeneous Continual Learning

**Authors:** Divyam Madaan, Hongxu Yin, Wonmin Byeon, Jan Kautz, Pavlo Molchanov  
**arXiv:** [2306.08593](https://arxiv.org/abs/2306.08593)  
**Date:** 2023-06-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.08593v1/

## Key points

- The authors propose a framework called Heterogeneous Continual Learning (HCL) for addressing continual learning with changing network architectures.
- HCL introduces evolving network architectures that emerge continually with novel data and tasks.
- HCL builds on the distillation family of techniques and modifies it for this new setting.
- In HCL, a weaker model acts as a teacher while a stronger architecture acts as a student, allowing for knowledge transfer between different architectures.
- Quick Deep Inversion (QDI) is proposed as a solution to recover prior task visual features to support knowledge transfer in scenarios with limited access to previous data.
- QDI reduces computational costs compared to previous solutions and improves overall performance.
- Evaluation on various benchmarks shows that HCL with modified knowledge distillation and QDI achieves significant improvements in accuracy compared to state-of-the-art methods across different network architectures.

## Summary

In this paper, the authors propose a novel framework called Heterogeneous Continual Learning (HCL) to address the problem of continual learning (CL) with changing network architectures. While most CL methods focus on adapting a single architecture to a new task by modifying its weights, the rapid progress in architecture design necessitates adapting existing solutions to novel architectures. To tackle this limitation, HCL introduces a wide range of evolving network architectures that emerge continually together with novel data and tasks. The authors build on top of the distillation family of techniques and modify it for this new setting. In HCL, a weaker model takes on the role of a teacher while a new stronger architecture acts as a student. This approach allows for knowledge transfer between different architectures. Additionally, the authors consider a setup where there is limited access to previous data and propose Quick Deep Inversion (QDI) as a solution to recover prior task visual features to support knowledge transfer. QDI significantly reduces computational costs compared to previous solutions and improves overall performance. The evaluation of various benchmarks shows that HCL with modified knowledge distillation and QDI achieves significant improvements in accuracy compared to state-of-the-art methods across various network architectures. In summary, this paper presents Heterogeneous Continual Learning as a new setup for addressing CL with changing network architectures. It introduces modifications to the knowledge distillation paradigm and proposes Quick Deep Inversion as an efficient method for enhancing distillation. The experimental results demonstrate the effectiveness of these approaches in improving accuracy in comparison to existing methods.