# SpecMemo: Speculative Decoding is in Your Pocket

**Authors:** Selin Yildirim, Deming Chen  
**arXiv:** [2506.01986](https://arxiv.org/abs/2506.01986)  
**Date:** 2025-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2506.01986v1/

## Key points

- Significant advancements in speculative decoding techniques have shown promising speedups in large language model (LLM) tasks
- Speculative decoding involves generating multiple candidate tokens to drive overall speedup
- Deploying speculative decoding on memory-constrained devices like mobile GPUs poses a challenge
- SpecMemo is a device-aware inference engine designed to efficiently manage memory allocations on limited memory devices
- SpecMemo enables multi-turn chatbots to utilize speculative decoding effectively by modeling the memory footprint and determining minimum required memory budget
- SpecMemo balances minimizing redundant memory allocations for rejected candidate tokens while preserving competitive performance gains from speculation
- SpecMemo maintains 96% of overall throughput from speculative decoding on MT-Bench while reducing generation-memory by 65% on a single Nvidia Titan RTX GPU
- SpecMemo extends its capabilities to facilitate big-model inference by leveraging multiple constrained GPUs and implementing batched speculative decoding approach for enhanced usability across small server GPUs
- The innovative framework has demonstrated a 2x speedup over distributed and batched vanilla decoding with the base model on eight AMD MI250 GPUs, leading to an 8x increase in inference throughput with a batch size of 10
- Overall, this work contributes towards democratizing LLM applications in resource-constrained environments and paves the way for faster and more cost-effective deployment of real-world LLM applications with robust performance

## Summary

In recent years, significant advancements in speculative decoding techniques have shown promising speedups in various large language model (LLM) tasks. These techniques involve generating multiple candidate tokens at the cost of extra memory allocations, with the acceptance rate of these candidates driving the overall speedup. However, deploying speculative decoding on memory-constrained devices like mobile GPUs poses a challenge in real-world scenarios. To address this issue, a device-aware inference engine called SpecMemo has been introduced to efficiently manage memory allocations on limited memory devices. This engine is designed to enable multi-turn chatbots to utilize speculative decoding effectively. The methodology behind SpecMemo involves theoretically modeling the memory footprint of speculative decoding to determine the minimum required memory budget while maintaining speedup. Through empirical testing, SpecMemo strikes a balance between minimizing redundant memory allocations for rejected candidate tokens and preserving competitive performance gains from speculation. Notably, with its capabilities in managing memory usage, SpecMemo maintains 96% of overall throughput from speculative decoding on MT-Bench while reducing generation-memory by 65% on a single Nvidia Titan RTX GPU. Furthermore, by leveraging multiple constrained GPUs, SpecMemo extends its capabilities to facilitate big-model inference by distributing the Llama-2-70B-Chat model. A novel batched speculative decoding approach has been implemented to enhance usability across multiple small server GPUs. This innovative framework has demonstrated a 2x speedup over distributed and batched vanilla decoding with the base model on eight AMD MI250 GPUs. Additionally, there has been a remarkable 8x increase in inference throughput with a batch size of 10. Overall, this work contributes towards democratizing LLM applications in resource-constrained environments and paves the way for faster and more cost-effective deployment of real-world LLM applications with robust performance.