# Structured Pruning Adapters

**Authors:** Lukas Hedegaard, Aman Alok, Juby Jose, Alexandros Iosifidis  
**arXiv:** [2211.10155](https://arxiv.org/abs/2211.10155)  
**Date:** 2022-11-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.10155v3/

## Key points

- Introduction of Structured Pruning Adapters (SPAs), a family of compressing, task-switching network adapters
- SPAs accelerate and specialize networks using tiny parameter sets and structured pruning
- Evaluation of channel-based SPA with various pruning methods on multiple computer vision benchmarks
- Channel-SPAs improve accuracy by an average of 6.9% while using only half the parameters at 90% pruned weights
- Channel-SPAs can learn adaptations with 17 times fewer parameters at 70% pruning with a slight decrease in accuracy of 1.6%
- Block-SPA requires significantly fewer parameters than pruning with fine-tuning
- Knowledge distillation using unpruned model as teacher helps retain accuracy better in pruning methods
- Other approaches for accelerating neural networks mentioned: Continual Inference Networks, quantization approaches, and pruning methods that remove unnecessary network weights from pre-trained models
- Structured Pruning Adapters offer an efficient alternative to fine-tuning by achieving higher accuracy with fewer parameters
- Experimental code and Python library of adapters available for further exploration

## Summary

The paper introduces Structured Pruning Adapters (SPAs), a family of compressing, task-switching network adapters that accelerate and specialize networks using tiny parameter sets and structured pruning. The authors propose a channel-based SPA and evaluate it with various pruning methods on multiple computer vision benchmarks. Compared to regular structured pruning with fine-tuning, the channel-SPAs improve accuracy by an average of 6.9% while using only half the parameters at 90% pruned weights. Alternatively, they can learn adaptations with 17 times fewer parameters at 70% pruning with a slight decrease in accuracy of 1.6%. Similarly, the block-SPA requires significantly fewer parameters than pruning with fine-tuning. The authors also mention that knowledge distillation using the unpruned model as the teacher has been found to help pruning methods retain accuracy better. The paper highlights other approaches for accelerating neural networks such as Continual Inference Networks which optimize computational sequences and intra-layer caching for online stream processing; quantization approaches that reduce model size and run-time costs through low-resolution numerical representations of network weights; and pruning methods that entirely remove unnecessary network weights from pre-trained models. Overall, the proposed Structured Pruning Adapters offer an efficient alternative to fine-tuning by achieving higher accuracy with fewer parameters. The experimental code and Python library of adapters are available for further exploration.