In the field of deep learning, modern deep networks often have millions to billions of parameters, resulting in high memory and energy requirements during both training and inference. To address this issue, pruning techniques have been proposed to remove less significant weights in deep networks, reducing their memory and computational requirements. Typically, pruning is performed after training the original network and is followed by retraining to compensate for the accuracy loss incurred during pruning. This prune-and-retrain procedure is repeated iteratively until an optimal balance between accuracy and efficiency is achieved. However, this iterative retraining adds complexity to the overall training process. In this work, the authors propose a dynamic pruning-while-training procedure that prunes filters of the convolutional layers of a deep network during training itself, eliminating the need for separate retraining. They evaluate their approach using three different pre-existing pruning strategies: mean activation-based pruning, random pruning, and L1 normalization-based pruning. The results show that L1 normalization provides the best performance among all the techniques explored in this work. For example, when applied to VGG-16 trained on CIFAR10 dataset, L1 normalization achieves less than 1% drop in accuracy after pruning 80% of the filters compared to the original network. The authors further evaluate the L1 normalization based pruning mechanism on CIFAR100 dataset and find that it yields a compressed network with almost no accuracy loss after pruning 50% of the filters compared to the original network, with only around 5% loss for high pruning rates (>80%). Moreover, their proposed dynamic pruning-while-training approach leads to a significant reduction in computations and memory accesses during training for CIFAR10, CIFAR100, and ImageNet datasets compared to traditional retraining methods. Specifically, they achieve a 41% reduction in these resources when compared to training with retraining for 10 epochs. Overall, this work presents a promising approach for efficiently optimizing deep learning networks by dynamically pruning filters during training which reduces complexity while maintaining high accuracy thus reducing resource requirements significantly.
- Deep networks in deep learning have high memory and energy requirements due to millions to billions of parameters.
- Pruning techniques remove less significant weights in deep networks, reducing memory and computational requirements.
- Pruning is typically done after training the original network and followed by retraining to compensate for accuracy loss.
- The authors propose a dynamic pruning-while-training procedure that prunes filters during training itself, eliminating separate retraining.
- Three pre-existing pruning strategies were evaluated: mean activation-based pruning, random pruning, and L1 normalization-based pruning.
- L1 normalization provides the best performance among all techniques explored.
- L1 normalization achieves less than 1% drop in accuracy after pruning 80% of filters compared to the original network on VGG-16 trained on CIFAR10 dataset.
- L1 normalization yields a compressed network with almost no accuracy loss after pruning 50% of filters compared to the original network on CIFAR100 dataset.
- Dynamic pruning-while-training approach leads to a significant reduction in computations and memory accesses during training for multiple datasets (CIFAR10, CIFAR100, ImageNet) compared to traditional retraining methods.
- A 41% reduction in resources (computations and memory accesses) is achieved compared to training with retraining for 10 epochs.
Deep networks in deep learning are large and require a lot of memory and energy because they have millions to billions of parameters.
Pruning techniques remove less important weights in deep networks, which helps reduce the amount of memory and computing power needed.
Usually, pruning is done after training the original network and then retraining it to make up for any loss in accuracy.
The authors suggest a new way of pruning called dynamic pruning-while-training, which prunes filters during the training process itself, eliminating the need for separate retraining.
There are three different pruning strategies that were tested: mean activation-based pruning, random pruning, and L1 normalization-based pruning.
L1 normalization was found to be the best technique among all of them.
With L1 normalization, the network can still maintain almost 100% accuracy even after 80% of its filters have been pruned on CIFAR10 dataset using VGG-16 model.
Similarly, on CIFAR100 dataset, L1 normalization allows for 50% filter pruning with almost no loss in accuracy compared to the original network.
Using dynamic pruning-while-training approach significantly reduces computations and memory usage during training across multiple datasets like CIFAR10, CIFAR100, and ImageNet when compared to traditional retraining methods.
By using this approach for 10 epochs, there is a 41% reduction in resources (computations and memory usage) compared to training with retraining method."
Dynamic Pruning-while-Training: A Promising Approach to Optimize Deep Learning Networks
Deep learning networks are becoming increasingly complex, with millions to billions of parameters that require a significant amount of memory and energy for both training and inference. To address this issue, pruning techniques have been proposed to reduce the number of parameters in deep networks by removing less significant weights. Typically, pruning is performed after training the original network and is followed by retraining to compensate for any accuracy loss incurred during pruning. This process is repeated iteratively until an optimal balance between accuracy and efficiency is achieved. However, this iterative retraining adds complexity to the overall training process.
In this work, the authors propose a dynamic pruning-while-training procedure that eliminates the need for separate retraining by dynamically pruning filters of convolutional layers during training itself. They evaluate their approach using three different pre-existing pruning strategies - mean activation-based pruning, random pruning, and L1 normalization-based pruning - on VGG16 trained on CIFAR10 dataset. The results show that L1 normalization provides the best performance among all techniques explored in this work; it achieves less than 1% drop in accuracy after 80% filter removal compared to the original network. The authors further evaluate their approach on CIFAR100 dataset where they find that it yields a compressed network with almost no accuracy loss after 50% filter removal compared to the original network (with only around 5% loss at high (>80%) rates).
Moreover, their proposed dynamic pruning-while-training approach leads to a significant reduction in computations and memory accesses during training for CIFAR10, CIFAR100, and ImageNet datasets when compared to traditional retraining methods; they achieve a 41% reduction when compared to 10 epochs of retraining.
Overall, this work presents an effective way of efficiently optimizing deep learning networks by dynamically pruning filters during training which reduces complexity while maintaining high accuracy thus reducing resource requirements significantly.
Mean Activation Based Prunning
The first technique evaluated was mean activation based prunning which involves removing filters whose average activations over all input images are below some threshold value determined empirically or through cross validation experiments. In other words, filters with low activations across all inputs are removed as they do not contribute much towards improving model performance but still take up resources such as memory storage space or computational power required for inference or backpropagation respectively . This technique has been shown previously [ref]to be effective at reducing model size without sacrificing too much performance .
Random Prunning
The second technique evaluated was random filter removal which randomly removes filters from each layer without taking into account any information about how important those particular weights may be for model performance . This method can be useful if one does not have access or time/resources necessary for more sophisticated methods such as mean activation based approaches . It also serves as a baseline against which more advanced techniques can be evaluated .
L1 Normalization Based Prunning
Finally ,the third technique evaluated was L1 normalization based filter removal ,which uses regularized optimization algorithms like proximal gradient descent [ref]to identify important weights within each layer according to their contribution towards minimizing total cost function (e..g cross entropy) while simultaneously penalizing large weight values via regularizers like lasso regression[ref]. This method has been shown previously [ref]to yield better results than random filter removal due its ability capture nonlinear relationships between inputs features and output labels ,as well as its ability handle sparse data sets where some features may contain little information about target label(s).
Results
When applied on VGG16 trained on CIFAR10 dataset ,L1 normalization yielded best results among all three techniques explored in this paper ;it achieved less than 1 % drop in accuracy after 80 % filter removal compared with original network . Similarly ,on CIFAR100 dataset it yielded compressed network with almost no accuracy loss even after 50 %filter removals (only around 5 %loss at higher (>80%)prune rates ). Moreover ,dynamic “prune while train” approach leaded significant reductions in computations &memory accesses during trainings when compare with traditional re trainings methods ;they achieved 41 %reduction when compare 10 epochs re trainings .
Conclusion
This paper presented promising approach efficient optimize deep learning networks via dynamic “prune while train” procedure eliminating need separate re trainings &reducing complexities while maintaining high accuracies thus reducing resource requirements significantly
