# What Matters in Transformers? Not All Attention is Needed

**Authors:** Shwai He, Guoheng Sun, Zheyu Shen, Ang Li  
**arXiv:** [2406.15786](https://arxiv.org/abs/2406.15786)  
**Date:** 2024-06-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.15786v1/

## Key points

- Study conducted by Shwai He, Guoheng Sun, Zhenyu Shen, and Ang Li from the University of Maryland, College Park
- Focus on Transformer-based large language models (LLMs) and scalability
- Aim to explore redundancy variability in Transformers and propose methods for enhancing model efficiency
- Use similarity-based metric to identify redundant structures based on output similarity to inputs
- Surprising findings regarding excessive similarity of attention layers in LLMs
- Insights provide guidance for future network architecture design for creating more compact yet efficient models
- Code available at https://github.com/Shwai-He/LLM-Drop for further exploration and implementation

## Summary

In the study "What Matters in Transformers? Not All Attention is Needed," conducted by Shwai He, Guoheng Sun, Zhenyu Shen, and Ang Li from the University of Maryland, College Park, the researchers delve into the realm of Transformer-based large language models (LLMs) and their scalability. The study aims to explore the variability of redundancy across different components within Transformers and proposes methods for pruning attention layers and implementing joint layer dropping techniques to enhance model efficiency. Using a similarity-based metric to identify redundant structures based on their output similarity to inputs, surprising findings are revealed regarding the excessive similarity of attention layers in LLMs. These insights provide valuable guidance for future network architecture design in creating more compact yet efficient models. The code for this research will be made available at https://github.com/Shwai-He/LLM-Drop for further exploration and implementation in future studies.