# Optimizing Speculative Decoding for Serving Large Language Models Using Goodput

**Authors:** Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, Hao Zhang  
**arXiv:** [2406.14066](https://arxiv.org/abs/2406.14066)  
**Date:** 2024-06-20  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2406.14066v1/

## Key points

- Speculative decoding (SD) is a highly effective technique in reducing inference latency for large language models (LLMs)
- SD differs from traditional methods by using proxies to predict potential outputs instead of having the LLM generate all tokens directly
- Implementing SD in real-time online LLM serving systems may not always result in latency improvements, especially under higher request rates or low speculation accuracy
- SmartSpec is a dynamic framework that assesses and determines the best speculation length for each request based on a metric called goodput, considering system load and speculation accuracy
- SmartSpec consistently reduces average request latency by up to 3.2 times compared to non-speculative decoding baselines across diverse scenarios involving different target models, draft models, request rates, and datasets
- SmartSpec's versatility extends to various styles of speculative decoding approaches including traditional model-based methods as well as model-free techniques like prompt lookup and tree-style decoding
- Optimizing speculative decoding through frameworks like SmartSpec can enhance the efficiency and performance of large language models in real-world applications across different workloads and system loads

## Summary

, , , , 

In the realm of large language models (LLMs), reducing inference latency is a critical goal, with speculative decoding (SD) emerging as a highly effective technique. SD differs from traditional methods by utilizing proxies to predict potential outputs instead of having the LLM generate all tokens directly. This approach allows for verification by the LLM without compromising generation quality. However, implementing SD in real-time online LLM serving systems, especially those with continuous batching, does not always result in latency improvements. In fact, under higher request rates or low speculation accuracy, it can paradoxically increase latency. Additionally, determining the optimal speculation length for different workloads under varying system loads remains a challenge. To address these issues, a dynamic framework called SmartSpec has been developed. SmartSpec dynamically assesses and determines the best speculation length for each request based on a novel metric called goodput. Goodput takes into account the current observed load of the entire system and speculation accuracy to optimize speculative execution costs. The effectiveness of SmartSpec is demonstrated through consistent reductions in average request latency by up to 3.2 times compared to non-speculative decoding baselines across diverse scenarios involving different target models, draft models, request rates, and datasets. Furthermore, SmartSpec's versatility extends to various styles of speculative decoding approaches including traditional model-based methods as well as model-free techniques such as prompt lookup and tree-style decoding. By comparing SmartSpec against vanilla auto-regressive inference and fixed-length speculative decoding baselines across workloads like online chatting, text-to-SQL, summarization, and question answering given context, its superiority becomes evident. In conclusion, optimizing speculative decoding through frameworks like SmartSpec holds promise for enhancing the efficiency and performance of large language models in real-world applications across a range of workloads and system loads.