Search Results for « Model pruning »

Federated learning (FL) allows model training from local data collected by edge/mobile devices while preserving data privacy, which has wide applicability to image and vision applications. A challenge is that client devices in FL usually have much more limited computation and communication resources compared to servers in a data center. To overcome this challenge, we propose PruneFL -a novel FL approach with adaptive and distributed parameter pruning, which adapts the model size during FL to reduce both communication and computation overhead and minimize the overall training time, while maintaining a similar accuracy as the original model. PruneFL includes initial pruning at a selected client and further pruning as part of the FL process. The model size is adapted during this process, which includes maximizing the approximate empirical risk reduction divided by the time of one FL round. Our experiments with various datasets on edge devices (e.g., Raspberry Pi) show that: 1) we significantly reduce the training time compared to conventional FL and various other pruning-based methods and 2) the pruned model with automatically determined size converges to an accuracy that is very similar to the original model, and it is also a lottery ticket of the original model.

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.

We study the neural network (NN) compression problem, viewing the tension between the compression ratio and NN performance through the lens of rate-distortion theory. We choose a distortion metric that reflects the effect of NN compression on the model output and derive the tradeoff between rate (compression) and distortion. In addition to characterizing theoretical limits of NN compression, this formulation shows that \emph{pruning}, implicitly or explicitly, must be a part of a good compression algorithm. This observation bridges a gap between parts of the literature pertaining to NN and data compression, respectively, providing insight into the empirical success of model pruning. Finally, we propose a novel pruning strategy derived from our information-theoretic formulation and show that it outperforms the relevant baselines on CIFAR-10 and ImageNet datasets.

Multidisciplinary initiatives in the new world of coronavirus were combined to limit the spread of the pandemic. Interestingly, the AI group was a part of those efforts. This result-based approach is used to help scan, assess, predict and track current patients and possibly potential patients. Developments for tracking social distances or recognizing face masks have made headlines in particular. Most current advanced approaches to face mask recognition are built based on deep learning which is dependent on a large number of face samples. Nearly everybody wears a mask during corona virus outbreak in order to effectively avoid the spread of COVID-19 virus. Our goal is to train a customized deep learning model that helps to detect even if or not a person wears a mask and study the concept of model pruning with Keras-Surgeon. Model pruning can be efficient in reducing model size, so that it can be easily implemented and inferred on embedded systems.

Abstract Deep convolutional neural networks deploy in applications are hindered by their large parameters and high computational cost. In this paper, we test different gradient optimizer’s effect on YOLOv3 model pruning. This is achieved by enforcing channel-level sparsity in YOLOv3 network. The model trained with Adam optimizer get 5× reduction in model size only after training 60 epochs.

Federated Learning (FL) is a privacy-preserving distributed deep learning paradigm that involves substantial communication and computation effort, which is a problem for resource-constrained mobile and IoT devices. Model pruning/sparsification develops sparse models that could solve this problem, but existing sparsification solutions cannot satisfy at the same time the requirements for low bidirectional communication overhead between the server and the clients, low computation overhead at the clients, and good model accuracy, under the FL assumption that the server does not have access to raw data to fine-tune the pruned models. We propose Complement Sparsification (CS), a pruning mechanism that satisfies all these requirements through a complementary and collaborative pruning done at the server and the clients. At each round, CS creates a global sparse model that contains the weights that capture the general data distribution of all clients, while the clients create local sparse models with the weights pruned from the global model to capture the local trends. For improved model performance, these two types of complementary sparse models are aggregated into a dense model in each round, which is subsequently pruned in an iterative process. CS requires little computation overhead on the top of vanilla FL for both the server and the clients. We demonstrate that CS is an approximation of vanilla FL and, thus, its models perform well. We evaluate CS experimentally with two popular FL benchmark datasets. CS achieves substantial reduction in bidirectional communication, while achieving performance comparable with vanilla FL. In addition, CS outperforms baseline pruning mechanisms for FL.

The deployment constraints in practical applications necessitate the pruning of large-scale deep learning models, i.e., promoting their weight sparsity. As illustrated by the Lottery Ticket Hypothesis (LTH), pruning also has the potential of improving their generalization ability. At the core of LTH, iterative magnitude pruning (IMP) is the predominant pruning method to successfully find 'winning tickets'. Yet, the computation cost of IMP grows prohibitively as the targeted pruning ratio increases. To reduce the computation overhead, various efficient 'one-shot' pruning methods have been developed, but these schemes are usually unable to find winning tickets as good as IMP. This raises the question of how to close the gap between pruning accuracy and pruning efficiency? To tackle it, we pursue the algorithmic advancement of model pruning. Specifically, we formulate the pruning problem from a fresh and novel viewpoint, bi-level optimization (BLO). We show that the BLO interpretation provides a technically-grounded optimization base for an efficient implementation of the pruning-retraining learning paradigm used in IMP. We also show that the proposed bi-level optimization-oriented pruning method (termed BiP) is a special class of BLO problems with a bi-linear problem structure. By leveraging such bi-linearity, we theoretically show that BiP can be solved as easily as first-order optimization, thus inheriting the computation efficiency. Through extensive experiments on both structured and unstructured pruning with 5 model architectures and 4 data sets, we demonstrate that BiP can find better winning tickets than IMP in most cases, and is computationally as efficient as the one-shot pruning schemes, demonstrating 2-7 times speedup over IMP for the same level of model accuracy and sparsity.

Model pruning methods reduce memory requirements and inference time of large-scale pre-trained language models after deployment.However, the actual pruning procedure is computationally intensive, involving repeated training and pruning until the required sparsity is achieved.This paper combines data pruning with movement pruning for Neural Machine Translation (NMT) to enable efficient fine-pruning.We design a dataset pruning strategy by leveraging cross-entropy scores of individual training instances.We conduct pruning experiments on the task of machine translation from Romanian-to-English and Turkish-to-English, and demonstrate that selecting hardto-learn examples (top-k) based on training cross-entropy scores outperforms other dataset pruning methods.We empirically demonstrate that data pruning reduces the overall steps required for convergence and the training time of movement pruning.Finally, we perform a series of experiments to tease apart the role of training data during movement pruning and uncover new insights to understand the interplay between data and model pruning in the context of NMT.

Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications.In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression.TextPruner offers structured posttraining pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks.We also propose a self-supervised pruning method that can be applied without the labeled data.Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model. 1

Federated Learning serves as a distributed framework for machine learning. Traditional approaches to federated learning often assume the independence and identical distribution (IID) of client data. However, real-world scenarios frequently feature personalized characteristics in client data, deviating from the IID assumption. Additionally, challenges such as substantial communication overhead and limited resources at edge nodes hinder the practical implementation of federated learning. In response to the challenges in deploying federated learning, including uneven data distribution, communication bottlenecks, and resource limitations at edge nodes, this paper introduces an individualized federated learning framework based on model pruning. This framework effectively adapts the client's local model to the personalized distribution of local data while meeting the model aggregation requirements on the server. Utilizing sparse operations, the framework achieves personalized model pruning, efficiently compresses model parameters, and reduces computational load on edge nodes. Presently, our approach demonstrates a compression ratio of 3.8% on the non-IID dataset Feminist without compromising final training accuracy, resulting in a 12.3% acceleration in training speed.