# Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding

**Authors:** Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, Carole-Jean Wu  
**arXiv:** [2404.16710](https://arxiv.org/abs/2404.16710)  
**Date:** 2024-04-25  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2404.16710v1/

## Key points

- Authors introduce LayerSkip as an end-to-end solution to accelerate inference of large language models (LLMs)
- LayerSkip combines techniques like layer dropout during training and self-speculative decoding during inference
- During training, layer dropout with varying rates for different layers and early exit loss mechanism are implemented
- Self-speculative decoding technique is proposed for inference, exiting at early layers and validating using remaining layers
- Implementation of LayerSkip demonstrates significant speedups in tasks like summarization, coding, and semantic parsing
- Experiments validate the effectiveness of LayerSkip in optimizing inference efficiency without compromising accuracy

## Summary

, , , , 

In the paper "Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding," authors Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A Aly, Beidi Chen, and Carole-Jean Wu introduce LayerSkip as an end-to-end solution to accelerate inference of large language models (LLMs). LayerSkip is a comprehensive strategy that combines innovative techniques like layer dropout during training and self-speculative decoding during inference to optimize the efficiency of LLMs. During training, the authors implement layer dropout with varying rates for different layers and an early exit loss mechanism where all transformer layers share a common exit. This approach enhances the accuracy of early exits at earlier layers without introducing additional modules to the model. Furthermore, during inference, the authors propose a novel self-speculative decoding technique where they exit at early layers and validate and rectify using the remaining layers of the model. This method boasts a reduced memory footprint compared to other speculative decoding methods and leverages shared compute resources and activations between draft and verification stages. The study includes experiments on various Llama model sizes across different training scenarios such as pretraining from scratch, continual pretraining, finetuning on specific data domains, and finetuning on specific tasks. The implementation of their inference solution demonstrates significant speedups in tasks like summarization for CNN/DM documents (up to 2.16x), coding (1.82x), and TOPv2 semantic parsing task (2.0x). Additionally, the authors conduct experiments to evaluate their training recipe under different types of training conditions including continual pretraining on a diverse dataset containing 52B tokens. The results showcase promising outcomes that validate the effectiveness of their approach. Overall, "Layer Skip" presents a comprehensive strategy for optimizing inference in large language models through innovative techniques like layer dropout during training and self-speculative decoding during inference. The findings highlight substantial improvements in efficiency without compromising accuracy across various tasks and datasets.