# Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

**Authors:** Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, Beidi Chen  
**arXiv:** [2402.12374](https://arxiv.org/abs/2402.12374)  
**Date:** 2024-02-19  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.12374v2/

## Key points

- Sequoia is an algorithm designed for efficient inference with large language models (LLMs)
- Speculative decoding is a promising approach to accelerate inference, but existing methods have limitations in scaling and adapting
- Sequoia introduces a dynamic programming algorithm to optimize tree structure for speculated tokens, enhancing scalability and efficiency
- Incorporates a novel sampling and verification method that improves speculative performance across various decoding temperatures
- Features a hardware-aware tree optimizer that selects optimal token tree size and depth based on specific hardware platform
- Evaluation results show significant speed improvements for large language models like Llama2-7B, Llama2-13B, and Vicuna-33B on A100
- Achieves impressive latency improvements in offloading settings on L40, with as low as 0.56 s/token for exact Llama2-70B inference latency

## Summary

, , , , 


Sequoia is a groundbreaking algorithm designed to address the challenges of efficient inference with large language models (LLMs). As the demand for LLMs continues to grow, the need for faster and more scalable inference methods becomes increasingly crucial. Speculative decoding has emerged as a promising approach to accelerate inference, but existing methods have limitations in scaling to larger speculation budgets and adapting to different hyperparameters and hardware configurations. In response to these challenges, Sequoia introduces a dynamic programming algorithm that optimizes the tree structure for speculated tokens, enhancing scalability and efficiency. Additionally, Sequoia incorporates a novel sampling and verification method that significantly improves speculative performance across various decoding temperatures, outperforming previous approaches. One of the key innovations of Sequoia is its hardware-aware tree optimizer, which automatically selects the optimal token tree size and depth based on the specific hardware platform. This feature maximizes speculative performance and ensures optimal utilization of computational resources. Evaluation results demonstrate the effectiveness of Sequoia in enhancing decoding speed for large language models such as Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to $4.04\times$, $3.73\times$, and $2.27\times$ respectively. In offloading settings on L40, Sequoia achieves impressive latency improvements with as low as 0.56 s/token for exact Llama2-70B inference latency, representing a significant advancement over existing systems. The collaborative efforts of authors Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen have resulted in the development of Sequoia: a scalable, robust, and hardware-aware solution for speculative decoding that pushes the boundaries of efficiency in large language model inference tasks.