# Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models

**Authors:** Georgy Tyukin, Gbetondji J-S Dovonon, Jean Kaddour, Pasquale Minervini  
**arXiv:** [2407.15516](https://arxiv.org/abs/2407.15516)  
**Date:** 2024-07-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.15516v1/

## Key points

- Authors address the increasing demand for Large Language Models (LLMs) and the challenge of serving them with low latencies due to quadratic input length complexity of attention layers.
- The study investigates the impact of dropping Multi-Layer Perceptron (MLP) and attention layers at inference time on Llama-v2 model performance.
- Findings reveal that selectively dropping deeper attention layers results in only a marginal decrease in performance but leads to significant speedups compared to dropping entire layers.
- Removing 33% of attention layers from a 13B Llama2 model causes a mere 1.8% drop in average performance over the OpenLLM benchmark.
- Skipping layers except for the latter ones leads to reduced performances as more layers are skipped, except when it comes to skipping attention layers.
- Selective layer dropping can offer substantial speed improvements without compromising overall model performance, shedding light on practical strategies for optimizing LLMs during inference.

## Summary

In their paper titled "Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models," authors Georgy Tyukin, Gbetondji J-S Dovonon, Jean Kaddour, and Pasquale Minervini address the increasing demand for Large Language Models (LLMs) in recent months. The challenge lies in serving these models with low latencies due to the quadratic input length complexity of attention layers. To tackle this issue, the authors investigate the impact of dropping Multi-Layer Perceptron (MLP) and attention layers at inference time on Llama-v2 model performance. Their findings reveal that selectively dropping deeper attention layers results in only a marginal decrease in performance but leads to significant speedups compared to dropping entire layers. For instance, removing 33% of attention layers from a 13B Llama2 model causes a mere 1.8% drop in average performance over the OpenLLM benchmark. Additionally, the study shows that skipping layers except for the latter ones leads to reduced performances as more layers are skipped, except when it comes to skipping attention layers. This research sheds light on practical strategies for optimizing LLMs during inference by emphasizing that selective layer dropping can offer substantial speed improvements without compromising overall model performance. By understanding how different components impact inference efficiency, researchers and practitioners can better tailor LLM architectures to meet real-world demands for fast and accurate natural language processing tasks.