# To prune, or not to prune: exploring the efficacy of pruning for model compression

**Authors:** Michael Zhu, Suyog Gupta  
**arXiv:** [1710.01878](https://arxiv.org/abs/1710.01878)  
**Date:** 2017-10-05  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1710.01878v1/

## Key points

- Study by Michael Zhu and Suyog Gupta on model pruning for model compression
- Model pruning aims to induce sparsity in deep neural networks by reducing nonzero-valued parameters
- Previous research showed deep networks can be pruned with minimal loss in accuracy
- Two approaches presented for model compression: traditional pruning techniques and gradual pruning technique
- Experiments on various neural network architectures show large-sparse models outperform small-dense models with up to 10x reduction in non-zero parameters
- Model pruning is an effective strategy for compressing deep neural networks without compromising performance

## Summary

In their study "To prune or not to prune: exploring the efficacy of pruning for model compression," Michael Zhu and Suyog Gupta investigate the use of model pruning as a technique to induce sparsity in deep neural networks. By reducing the number of nonzero-valued parameters in a model's connection matrices, model pruning aims to significantly decrease model size without sacrificing accuracy. Building on previous research by Han et al. (2015) and Narang et al. (2017), which showed that deep networks can be pruned with minimal loss in accuracy, Zhu and Gupta propose that baseline models may be excessively over-parameterized from the start. They present two approaches for model compression within the context of energy-efficient inference in resource-constrained environments. The first approach uses traditional pruning techniques to reduce hidden units while maintaining dense connections. The second approach introduces a new gradual pruning technique that can be easily applied across different models and datasets with minimal tuning requirements, seamlessly integrating it into the training process. Through experiments on various neural network architectures such as deep CNNs, stacked LSTM, and seq2seq LSTM models, Zhu and Gupta compare large-sparse models (pruned but large) with small-dense models (smaller but dense) having identical memory footprints. Surprisingly, they find that large-sparse models consistently outperform small-dense models while achieving up to 10x reduction in non-zero parameters with minimal loss in accuracy. This study highlights the effectiveness of model pruning as a viable strategy for compressing deep neural networks without compromising performance. By exploring different methods for model compression and introducing innovative techniques like gradual pruning, Zhu and Gupta provide valuable insights towards optimizing neural network efficiency in real-world applications where computational resources are limited.