# Directions of Curvature as an Explanation for Loss of Plasticity

**Authors:** Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, Marlos C. Machado  
**arXiv:** [2312.00246](https://arxiv.org/abs/2312.00246)  
**Date:** 2023-11-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.00246v4/

## Key points

- Loss of plasticity in neural networks is a phenomenon where networks struggle to learn from new experiences.
- The authors suggest that loss of plasticity occurs due to the loss of directions of curvature during training.
- Their research involved investigating continual learning tasks using datasets like MNIST, CIFAR-10, and ImageNet to support their claim.
- The study found that the loss of curvature directions correlates with the loss of plasticity in neural networks.
- The authors challenge previous explanations for loss of plasticity by providing counterexamples using a linearly separable subset of the MNIST dataset with periodically shuffled labels.
- They explore how regularizers can help mitigate loss of plasticity by preserving curvature in neural networks.
- A distributional regularizer introduced by the authors was effective in maintaining plasticity across different problem settings.

## Summary

In their paper "Directions of Curvature as an Explanation for Loss of Plasticity," authors Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C. Machado delve into the phenomenon of loss of plasticity in neural networks. This refers to the inability of neural networks to learn from new experiences, a problem that has been observed in various settings without a clear understanding of its underlying mechanisms. The authors propose a consistent explanation for this issue: during training, neural networks lose directions of curvature which leads to a reduction in plasticity. To support their claim, they conduct a systematic investigation across continual learning tasks using popular datasets such as MNIST, CIFAR-10, and ImageNet. Their findings demonstrate that the loss of curvature directions aligns with the loss of plasticity, highlighting the significance of this factor in neural network behavior. Moreover, the authors challenge previous explanations for loss of plasticity by providing counterexamples using a linearly separable subset of the MNIST dataset with periodically shuffled labels. This analysis reveals inconsistencies in existing explanations and emphasizes the complexities associated with preserving plasticity even in simple classification problems. Furthermore, the paper explores how regularizers can mitigate loss of plasticity by preserving curvature in neural networks. The authors introduce a distributional regularizer that proves effective across different problem settings considered in their study.