# The Unreasonable Ineffectiveness of the Deeper Layers

**Authors:** Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts  
**arXiv:** [2403.17887](https://arxiv.org/abs/2403.17887)  
**Date:** 2024-03-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.17887v1/

## Key points

- Study titled "The Unreasonable Ineffectiveness of the Deeper Layers" investigates layer pruning for Large Language Models (LLMs)
- Minimal degradation in performance observed by removing up to half of the layers on question-answering benchmarks
- Optimal block of layers for pruning identified using similarity assessment; performance loss mitigated through techniques like quantization and Low Rank Adapters (QLoRA)
- Experiments efficiently conducted on a single A100 GPU
- Layer pruning can complement other finetuning strategies, reducing computational resources during training and enhancing memory/latency efficiency during inference
- Robustness of LLMs to layer removal raises questions about the role of shallow layers in storing knowledge

## Summary

In their study titled "The Unreasonable Ineffectiveness of the Deeper Layers," authors Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts empirically investigate the effectiveness of layer pruning for popular families of open-weight pretrained Large Language Models (LLMs). By removing a significant fraction of layers (up to half), they observe minimal degradation in performance on various question-answering benchmarks. The researchers use similarity assessment to identify the optimal block of layers for pruning and mitigate any performance loss through parameter-efficient techniques such as quantization and Low Rank Adapters (QLoRA). Notably, these experiments are efficiently conducted on a single A100 GPU. From a practical standpoint, the results suggest that layer pruning can complement other parameter-efficient finetuning strategies, leading to further reductions in computational resources during training while enhancing memory and latency efficiency during inference. Moreover, from a scientific perspective, the robustness of LLMs to layer removal raises questions about the utilization of parameters in deeper layers versus the potential critical role played by shallow layers in storing knowledge. Overall, this research sheds light on the benefits of layer pruning techniques in optimizing computational resources and improving model efficiency without compromising performance on question-answering tasks. is an effective strategy for improving , as shown by  Their study reveals that  can be enhanced through layer removal without sacrificing performance on . Additionally, may need to consider the importance of shallow layers in storing knowledge.