# Latent Replay for Real-Time Continual Learning

**Authors:** Lorenzo Pellegrini, Gabrile Graffieti, Vincenzo Lomonaco, Davide Maltoni  
**arXiv:** [1912.01100](https://arxiv.org/abs/1912.01100)  
**Date:** 2019-12-02  
**Summary page:** https://www.summarizepaper.com/en/paper/1912.01100v1/

## Key points

- Training deep neural networks on lightweight computational devices is a persistent challenge
- Continual learning techniques have emerged as a potential solution, allowing complex models to be incrementally trained on small batches of new data
- Practical problems such as catastrophic forgetting need to be addressed
- The innovative technique called "Latent Replay" stores activation volumes at some intermediate layer instead of storing past data in the input space, significantly reducing computation and storage required by native rehearsal
- To ensure that the representation remains stable and stored activations remain valid, the researchers propose slowing down learning at all layers below the latent replay one while leaving layers above free to learn at full pace
- Pellegrini's team combined Latent Replay with existing continual learning techniques and achieved state-of-the-art accuracy on CORe50 NICv2 - a difficult benchmark consisting of nearly 400 small and highly non-i.i.d. batches
- They demonstrated that nearly real-time continual learning on the edge is feasible through porting their proposed technique onto a smartphone device
- This research offers an important contribution towards addressing challenges associated with training deep neural networks on lightweight computational devices, opening up new possibilities for real-time machine learning applications in resource-constrained environments.

## Summary

The challenge of training deep neural networks on lightweight computational devices has been a persistent issue. However, continual learning techniques have emerged as a potential solution, allowing complex models to be incrementally trained on small batches of new data, making the learning problem tractable even for CPU-only edge devices. Despite this, practical problems such as catastrophic forgetting need to be addressed. In response, a team of researchers led by Lorenzo Pellegrini have introduced an innovative technique called "Latent Replay". Instead of storing past data in the input space, Latent Replay stores activation volumes at some intermediate layer. This approach significantly reduces the computation and storage required by native rehearsal. To ensure that the representation remains stable and stored activations remain valid, the researchers propose slowing down learning at all layers below the latent replay one while leaving layers above free to learn at full pace. In their experiments, Pellegrini's team combined Latent Replay with existing continual learning techniques and achieved state-of-the-art accuracy on CORe50 NICv2 - a difficult benchmark consisting of nearly 400 small and highly non-i.i.d. batches. Furthermore, they demonstrated that nearly real-time continual learning on the edge is feasible through porting their proposed technique onto a smartphone device. Overall, this research offers an important contribution towards addressing the challenges associated with training deep neural networks on lightweight computational devices. By introducing Latent Replay as an innovative technique for reducing computation and storage requirements while maintaining model accuracy and stability during continual learning processes, Pellegrini's team has opened up new possibilities for real-time machine learning applications in resource-constrained environments.