AI summary · arXiv: 1810.05331v2 · cs.CV
Dynamic Channel Pruning: Feature Boosting and Suppression
The essentials
- Authors propose a method called feature boosting and suppression (FBS) to reduce computational and memory resources required for deep CNNs without sacrificing accuracy.
- FBS introduces small auxiliary connections to existing convolutional layers, allowing for dynamic amplification of salient convolutional channels and skipping of unimportant ones at runtime.
- FBS preserves the full network structures and accelerates convolution by dynamically skipping unimportant input and output channels, unlike permanent channel pruning methods.
- Experimental results show that FBS can provide up to 5 times savings in compute on VGG-16 and ResNet-18 models with less than 0.6% top-5 accuracy loss.
- FBS-augmented networks can be trained using conventional stochastic gradient descent, making it applicable to many state-of-the-art CNN architectures.
- This approach has potential implications for cost-sensitive cloud services and low-powered edge computing applications where reducing resource requirements is crucial.
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Yi-Hui He (2 papers)
5 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Understanding the Impact of Dynamic Channel Pruning on Conditionally Parameterized ConvolutionsRavi S. Raju, Dibakar Gope, Urmish Thakker et al. · 2020 · AIChallengeIoT@SenSys · cited 4 times · Not summarized yet
Understanding the Impact analyzes FBS’s runtime threshold gating and its effects on conditionally parameterized convolutions.
Abstract
Recent trends have shown that deep learning models have become larger and more accurate at an increased computational cost, making them difficult to deploy for latency-constrained applications. Conditional execution methods address this increase in cost while preserving the accuracy of the model by only performing the most important computation on important features. In this paper, we analyze a recent method, Feature Boosting and Suppression (FBS), which dynamically assesses which channels contain the most important input-dependent features and prune the others based on a runtime threshold gating mechanism. FBS is able to dynamically prune convolution filters with little loss in accuracy at …
-
Accelerating Convolutional Neural Networks with Dynamic Channel PruningChi-Liang Zhang, Tao Hu, Yingda Guan et al. · 2019 · Data Compression Conference · cited 21 times · Not summarized yet
Accelerating Convolutional Neural Networks with Dynamic Channel Pruning proposes channel pruning to reduce CNN computation, a problem also addressed by FBS.
Abstract
Network acceleration has become a hot topic, for the substantial challenge in deploying such networks in real-time applications or on resource-limited devices. A wide variety of pruning-based acceleration methods were proposed to expend the sparsity of parameters, thus omit computations involving those pruned parameters. However, these element-wise pruning methods can hardly be efficiently used for accelerating without special-customized speed-up algorithms. Due to this difficulty, recent work has turned to prune filters or channels instead, which directly reduce the number of matrix multiplications. While Channel Pruning method reforms the original CNNs to a kernel-wisely or channel-wisely …
-
A Channel Pruning Method Using Subtraction-Average-Based OptimizerZhou-Feng Liu, Wentao Wu, Shu-Min Ding et al. · 2024 · International Conference on Robotics, Intelligent Control and Artificial Intelligence · Not summarized yet
SABOPruner uses a subtraction-average-based optimizer to select channels for pruning, rather than FBS’s input-dependent runtime selection.
Abstract
Convolutional Neural Networks(CNNs) require huge resources, which hindered the implement on light devices. However, existing pruning techniques primarily concentrate on choosing channels based on their importance or on regularization strategies that are crafted through empirical knowledge, the compressed networks obtained through these methods have unstable performance and require unnecessary human intervention. This paper proposes a new channel pruning method based on the Subtraction-Average-Based Optimizer, dubbed as SABOPruner. To facilitate deployment on FGPA, we constrain the pruning step size of each layer to be a multiple of 8. Finally, the search was transformed into an optimization …
-
Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural NetworksZhouyang Xie, Yan Fu, Sheng-Zhao Tian et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
Pruning with Compensation proposes an efficient channel-pruning procedure to reduce deep CNN inference costs, unlike FBS’s dynamic channel skipping.
Abstract
Channel pruning is a promising technique to compress the parameters of deep convolutional neural networks(DCNN) and to speed up the inference. This paper aims to address the long-standing inefficiency of channel pruning. Most channel pruning methods recover the prediction accuracy by re-training the pruned model from the remaining parameters or random initialization. This re-training process is heavily dependent on the sufficiency of computational resources, training data, and human interference(tuning the training strategy). In this paper, a highly efficient pruning method is proposed to significantly reduce the cost of pruning DCNN. The main contributions of our method include: 1) pruning …
-
Pruning Very Deep Neural Network Channels for Efficient InferenceYi-Hui He · 2022 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
Pruning Very Deep Neural Network Channels selects channels using LASSO regression and least-squares reconstruction to accelerate CNN inference.
Abstract
—In this paper, we introduce a new channel pruning method to accelerate very deep convolutional neural networks. Given a trained CNN model, we propose an iterative two-step algorithm to effectively prune each layer, by a LASSO regression based channel selection and least square reconstruction. We further generalize this algorithm to multi-layer and multi-branch cases. Our method reduces the accumulated error and enhances the compatibility with various architectures. Our pruned VGG-16 achieves the state-of-the-art results by 5 × speed-up along with only 0.3% increase of error. More importantly, our method is able to accelerate modern networks like ResNet, Xception and suffers only 1.4%, 1.0% …
-
Boosting the Performance of CNN Accelerators with Dynamic Fine-Grained Channel GatingWeizhe Hua, Yuan Zhou, Christopher De Sa et al. · 2019 · Micro · cited 48 times · Not summarized yet
Dynamic Fine-Grained Channel Gating uses input-specific channel gating in feature-map regions and develops an accelerator to exploit that sparsity.
Abstract
This paper proposes a new fine-grained dynamic pruning technique for CNN inference, named channel gating, and presents an accelerator architecture that can effectively exploit the dynamic sparsity. Intuitively, channel gating identifies the regions in the feature map of each CNN layer that contribute less to the classification result and turns off a subset of channels for computing the activations in these less important regions. Unlike static network pruning, which removes redundant weights or neurons prior to inference, channel gating exploits dynamic sparsity specific to each input at run time and in a structured manner. To maximize compute savings while minimizing accuracy loss, channel …
-
Dynamic Neural Network Channel Execution for Efficient TrainingSimeon E. Spasov, P. Lio' · 2019 · British Machine Vision Conference · cited 6 times · Not summarized yet
Dynamic Neural Network Channel Execution selects salient channels during training as well as inference, extending channel execution to training efficiency.
Abstract
Existing methods for reducing the computational burden of neural networks at run-time, such as parameter pruning or dynamic computational path selection, focus solely on improving computational efficiency during inference. On the other hand, in this work, we propose a novel method which reduces the memory footprint and number of computing operations required for training and inference. Our framework efficiently integrates pruning as part of the training procedure by exploring and tracking the relative importance of convolutional channels. At each training step, we select only a subset of highly salient channels to execute according to the combinatorial upper confidence bound algorithm, and r…
-
CDCP: A Framework for Accelerating Dynamic Channel Pruning MethodsZhaohong Xiang, Yi-Gui Luo, Yin Xie et al. · 2023 · International Conference on Parallel and Distributed Systems · Not summarized yet
CDCP proposes a framework to accelerate dynamic channel-pruning methods, including the class of methods represented by FBS.
Abstract
Dynamic channel pruning is a technique aimed at reducing the theoretical computational complexity and inference latency of convolutional neural networks. Dynamic channel pruning methods introduce complex additional modules for dynamically selecting channels for images. Due to the additional modules, dynamic channel pruning methods never achieve optimal acceleration effect in real world. To address this problem, we propose Consecutive Dynamic Channel Pruning (CDCP), a novel dynamic channel pruning framework unified for almost all dynamic pruning methods designed for continuous image processing. The core idea of CDCP stems from our observation that adjusting the network for all frames in seman…
Prior work it builds on
-
Pruning Filters for Efficient ConvNetsHao Li, Asim Kadav, Igor Durdanovic et al. · 2016 · International Conference on Learning Representations · cited 4,297 times · Not summarized yet
Pruning Filters removes whole filters with small effects on output accuracy to accelerate CNNs, whereas FBS skips channels dynamically by input.
Abstract
The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy. However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in…
-
Discrimination-aware Channel Pruning for Deep Neural NetworksZhuangwei Zhuang, Mingkui Tan, Bo-Han Zhuang et al. · 2018 · Neural Information Processing Systems · cited 648 times · Not summarized yet
Discrimination-aware Channel Pruning selects channels for their contribution to discriminative power, rather than FBS’s input-dependent runtime salience.
Abstract
Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. To overcome these drawbacks, we investigate a simple-yet-effective method, called discrimination-aware channel pruning, to choose those channels that really contribute to discriminative power. To this end…
-
Soft Filter Pruning for Accelerating Deep Convolutional Neural NetworksYang He, Guoliang Kang, Xuanyi Dong et al. · 2018 · Computer Vision and Pattern Recognition · cited 1,121 times · AI summary available
Soft Filter Pruning updates pruned filters during training to accelerate CNN inference, while FBS dynamically skips channels at runtime.
Abstract
This paper proposed a Soft Filter Pruning (SFP) method to accelerate the inference procedure of deep Convolutional Neural Networks (CNNs). Specifically, the proposed SFP enables the pruned filters to be updated when training the model after pruning. SFP has two advantages over previous works: (1) Larger model capacity. Updating previously pruned filters provides our approach with larger optimization space than fixing the filters to zero. Therefore, the network trained by our method has a larger model capacity to learn from the training data. (2) Less dependence on the pre-trained model. Large capacity enables SFP to train from scratch and prune the model simultaneously. In contrast, previous…
-
ThiNet: A Filter Level Pruning Method for Deep Neural Network CompressionJian-Hao Luo, Jian-Xin Wu, Wei-Yao Lin · 2017 · IEEE International Conference on Computer Vision · cited 1,911 times · Not summarized yet
ThiNet prunes filters using statistics from the next layer to compress and accelerate CNNs, rather than selecting channels dynamically per input.
Abstract
We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this …
-
Learning to Prune Filters in Convolutional Neural NetworksQiangui Huang, Kevin Zhou, Su-Ya You et al. · 2018 · IEEE Workshop/Winter Conference on Applications of Computer Vision · cited 187 times · Not summarized yet
Learning to Prune Filters trains agents to remove unnecessary CNN filters, rather than using FBS’s runtime channel selection.
Abstract
Many state-of-the-art computer vision algorithms use large scale convolutional neural networks (CNNs) as basic building blocks. These CNNs are known for their huge number of parameters, high redundancy in weights, and tremendous computing resource consumptions. This paper presents a learning algorithm to simplify and speed up these CNNs. Specifically, we introduce a “try-and-learn” algorithm to train pruning agents that remove unnecessary CNN filters in a data-driven way. With the help of a novel reward function, our agents removes a significant number of filters in CNNs while maintaining performance at a desired level. Moreover, this method provides an easy control of the tradeoff between n…
Papers that cite it
-
BilevelPruning: Unified Dynamic and Static Channel Pruning for Convolutional Neural NetworksShangqian Gao, Yan-Fu Zhang, Feihu Huang et al. · 2024 · Computer Vision and Pattern Recognition · cited 29 times · Not summarized yet
BilevelPruning unifies dynamic and static channel pruning, and cites FBS as related work.
-
Multidimensional Dynamic Pruning: Exploring Spatial and Channel Fuzzy SparsityMingwen Shao, Jiandong Kuang, Chao Wang et al. · 2024 · IEEE transactions on fuzzy systems · cited 8 times · Not summarized yet
Multidimensional Dynamic Pruning extends dynamic pruning across channel and spatial dimensions and cites FBS.
Abstract
Dynamic pruning is an effective model compression method to reduce the computational cost of networks. However, existing dynamic pruning methods are limited to pruning along a single dimension (channel, spatial, or depth), which cannot maximally excavate the redundancy of the network. Meanwhile, most of the current state-of-the-arts usually implement dynamic pruning via masked-out partial channels and pixels for training while failing to accelerate the inference speed. To tackle these limitations, we propose a novel fuzzy-based multidimensional dynamic pruning paradigm to dynamically compress neural networks along both the channel and spatial dimensions. Specifically, we design a multidimens…
-
Self-Adaptive Network PruningJinting Chen, Zhaocheng Zhu, Chengwei Li et al. · 2019 · International Conference on Neural Information Processing · cited 24 times · Not summarized yet
Self-Adaptive Network Pruning predicts channel saliency per layer and sample under a computation budget, citing FBS among related methods.
Abstract
Deep convolutional neural networks have been proved successful on a wide range of tasks, yet they are still hindered by their large computation cost in many industrial scenarios. In this paper, we propose to reduce such cost for CNNs through a self-adaptive network pruning method (SANP). Our method introduces a general Saliency-and-Pruning Module (SPM) for each convolutional layer, which learns to predict saliency scores and applies pruning for each channel. Given a total computation budget, SANP adaptively determines the pruning strategy with respect to each layer and each sample, such that the average computation cost meets the budget. This design allows SANP to be more efficient in comput…
-
Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask PredictionSara Elkerdawy, Mostafa Elhoushi, Hong Zhang et al. · 2021 · Computer Vision and Pattern Recognition · cited 62 times · Not summarized yet
Fire Together Wire Together predicts dynamic pruning masks with self-supervision and cites FBS as an earlier dynamic pruning method.
Abstract
Dynamic model pruning is a recent direction that allows for the inference of a different sub-network for each input sample during deployment. However, current dynamic methods rely on learning a continuous channel gating through regularization by inducing sparsity loss. This formulation introduces complexity in balancing different losses (e.g task loss, regularization loss). In addition, regularization based methods lack transparent tradeoff hyper- parameter selection to realize a computational budget. Our contribution is two-fold: 1) decoupled task and pruning losses. 2) Simple hyperparameter selection that enables FLOPs reduction estimation before training. Inspired by the Hebbian theory in…
-
Progressive Channel-Shrinking NetworkJianhong Pan, Si-Yuan Yang, Lin Geng Foo et al. · 2023 · IEEE transactions on multimedia · cited 5 times · Not summarized yet
Progressive Channel-Shrinking addresses gating and indexing costs in dynamic channel pruning and cites FBS as prior work.
Abstract
Currently, salience-based channel pruning makes continuous breakthroughs in network compression. In the realization, the salience mechanism is used as a metric of channel salience to guide pruning. Therefore, salience-based channel pruning can dynamically adjust the channel width at run-time, which provides a flexible pruning scheme. However, there are two problems emerging: a gating function is often needed to truncate the specific salience entries to zero, which destabilizes the forward propagation; dynamic architecture brings more cost for indexing in inference which bottlenecks the inference speed. In this article, we propose a Progressive Channel-Shrinking (PCS) method to compress the s…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 4 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.