# Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions

**Authors:** Kaiqi Zhao, Animesh Jain, Ming Zhao  
**arXiv:** [2303.08595](https://arxiv.org/abs/2303.08595)  
**Date:** 2023-03-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.08595v1/

## Key points

- The paper introduces Automatic Attention Pruning (AAP) as a novel approach to compress deep learning models.
- AAP addresses the limitations of existing pruning solutions based on unstructured pruning.
- AAP is an adaptive, attention-based, structured pruning approach.
- AAP involves two key components: iterative structured pruning using activation-based attention maps and adaptive pruning policies.
- Activation-based attention maps are used to identify and prune unimportant filters in a deep learning model.
- Adaptive pruning policies automatically meet different pruning objectives based on specific requirements.
- AAP ensures small, accurate, and hardware-efficient models while meeting user objectives.
- Comprehensive evaluation shows that AAP outperforms existing structured pruning techniques significantly.
- AAP presents an innovative solution for automating and improving model compression for resource-constrained edge devices.

## Summary

The paper titled "Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions" introduces a novel approach called Automatic Attention Pruning (AAP) to address the limitations of existing pruning solutions for compressing deep learning models. These solutions are often based on unstructured pruning, which results in models that cannot efficiently run on commodity hardware. AAP is an adaptive, attention-based, structured pruning approach that aims to automatically generate small, accurate, and hardware-efficient models while meeting user objectives. The proposed method involves two key components: iterative structured pruning using activation-based attention maps and adaptive pruning policies. In the first component, AAP utilizes activation-based attention maps to identify and prune unimportant filters in a deep learning model. This iterative process helps effectively reduce model size without sacrificing accuracy. The second component focuses on adaptive pruning policies that automatically meet different pruning objectives based on specific requirements such as accuracy-critical tasks, memory-constrained scenarios, and latency-sensitive applications. By adapting the pruning process according to these objectives, AAP ensures that the resulting models are optimized for their intended use cases. To evaluate the effectiveness of AAP, a comprehensive evaluation was conducted comparing it with state-of-the-art structured pruning techniques across various model architectures. The results demonstrate that AAP outperforms existing methods significantly. In conclusion, Automatic Attention Pruning presents an innovative solution to automate and improve the model compression process for resource-constrained edge devices by leveraging attention-based techniques and adaptive policies.