# Pruning Filters while Training for Efficiently Optimizing Deep Learning Networks

**Authors:** Sourjya Roy, Priyadarshini Panda, Gopalakrishnan Srinivasan, Anand Raghunathan  
**arXiv:** [2003.02800](https://arxiv.org/abs/2003.02800)  
**Date:** 2020-03-05  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2003.02800v1/

## Key points

- Deep networks in deep learning have high memory and energy requirements due to millions to billions of parameters.
- Pruning techniques remove less significant weights in deep networks, reducing memory and computational requirements.
- Pruning is typically done after training the original network and followed by retraining to compensate for accuracy loss.
- The authors propose a dynamic pruning-while-training procedure that prunes filters during training itself, eliminating separate retraining.
- Three pre-existing pruning strategies were evaluated: mean activation-based pruning, random pruning, and L1 normalization-based pruning.
- L1 normalization provides the best performance among all techniques explored.
- L1 normalization achieves less than 1% drop in accuracy after pruning 80% of filters compared to the original network on VGG-16 trained on CIFAR10 dataset.
- L1 normalization yields a compressed network with almost no accuracy loss after pruning 50% of filters compared to the original network on CIFAR100 dataset.
- Dynamic pruning-while-training approach leads to a significant reduction in computations and memory accesses during training for multiple datasets (CIFAR10, CIFAR100, ImageNet) compared to traditional retraining methods.
- A 41% reduction in resources (computations and memory accesses) is achieved compared to training with retraining for 10 epochs.

## Summary

In the field of deep learning, modern deep networks often have millions to billions of parameters, resulting in high memory and energy requirements during both training and inference. To address this issue, pruning techniques have been proposed to remove less significant weights in deep networks, reducing their memory and computational requirements. Typically, pruning is performed after training the original network and is followed by retraining to compensate for the accuracy loss incurred during pruning. This prune-and-retrain procedure is repeated iteratively until an optimal balance between accuracy and efficiency is achieved. However, this iterative retraining adds complexity to the overall training process. In this work, the authors propose a dynamic pruning-while-training procedure that prunes filters of the convolutional layers of a deep network during training itself, eliminating the need for separate retraining. They evaluate their approach using three different pre-existing pruning strategies: mean activation-based pruning, random pruning, and L1 normalization-based pruning. The results show that L1 normalization provides the best performance among all the techniques explored in this work. For example, when applied to VGG-16 trained on CIFAR10 dataset, L1 normalization achieves less than 1% drop in accuracy after pruning 80% of the filters compared to the original network. The authors further evaluate the L1 normalization based pruning mechanism on CIFAR100 dataset and find that it yields a compressed network with almost no accuracy loss after pruning 50% of the filters compared to the original network, with only around 5% loss for high pruning rates (>80%). Moreover, their proposed dynamic pruning-while-training approach leads to a significant reduction in computations and memory accesses during training for CIFAR10, CIFAR100, and ImageNet datasets compared to traditional retraining methods. Specifically, they achieve a 41% reduction in these resources when compared to training with retraining for 10 epochs. Overall, this work presents a promising approach for efficiently optimizing deep learning networks by dynamically pruning filters during training which reduces complexity while maintaining high accuracy thus reducing resource requirements significantly.