# SD$^2$: Self-Distilled Sparse Drafters

**Authors:** Mike Lasby, Nish Sinnadurai, Valavan Manohararajah, Sean Lie, Vithursan Thangarasa  
**arXiv:** [2504.08838](https://arxiv.org/abs/2504.08838)  
**Date:** 2025-04-10  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2504.08838v1/

## Key points

- Speculative decoding in Large Language Models (LLMs) minimizes latency and improves efficiency
- Self-Distilled Sparse Drafters (SD$^2$) methodology utilizes self-data distillation and weight sparsity for crafting efficient draft models
- SD$^2$ boosts draft token acceptance rates and reduces Multiply-Accumulate operations (MACs)
- In the Universal Assisted Generation scenario, SD$^2$ excels even with different model families
- When applied to a Llama-3.1-70B target model, SD$^2$ increases Mean Accepted Length (MAL) by $\times$1.59 compared to layer-pruned draft models
- SD$^2$ achieves over 43.87% reduction in MACs with only an 8.36% decrease in MAL compared to dense draft models
- Sparsity-aware fine-tuning and compression strategies enhance LLM inference efficiency while maintaining alignment with target models
- Potential avenues for further exploration include integrating quantization-aware training, draft-token trees, compressed KV-cache implementations, and benchmarking on hardware optimized for unstructured sparsity
- Advanced fine-tuning techniques like speculative knowledge distillation or square-head distillation can elevate the quality of draft models generated using SD$^2$
- Combining structured pruning with fine-grained sparsity and quantization can reduce inference-time latency while upholding accuracy standards

## Summary

In the realm of Large Language Models (LLMs), speculative decoding has emerged as a potent tool for minimizing latency and improving efficiency. This allows for the utilization of highly compressed draft models within a fault-tolerant framework. Building upon this foundation, a groundbreaking methodology known as Self-Distilled Sparse Drafters (SD$^2$) has been introduced. This innovative approach harnesses the power of self-data distillation and fine-grained weight sparsity to craft exceptionally efficient and well-aligned draft models. SD$^2$ stands out by systematically boosting draft token acceptance rates while simultaneously slashing Multiply-Accumulate operations (MACs). Even in the challenging Universal Assisted Generation (UAG) scenario, where draft and target models stem from different model families, SD$^2$ excels. Notably, when applied to a Llama-3.1-70B target model, SD$^2$ delivers a remarkable $\times$1.59 increase in Mean Accepted Length (MAL) compared to layer-pruned draft models. Furthermore, it achieves an impressive reduction of over 43.87% in MACs with only an 8.36% decrease in MAL compared to dense draft models. These results underscore the immense potential of sparsity-aware fine-tuning and compression strategies in enhancing LLM inference efficiency while maintaining alignment with target models. Moreover, there are exciting avenues for further exploration and refinement stemming from this work. For instance, integrating quantization-aware training, draft-token trees, and compressed KV-cache implementations with SD$^2$ could bolster memory and compute efficiency significantly. Additionally, benchmarking SD$^2$ on hardware optimized for unstructured sparsity could yield valuable insights into its performance under different conditions. Incorporating advanced fine-tuning techniques like speculative knowledge distillation or square-head distillation may further elevate the quality of draft models generated using SD$^2$. Lastly, combining structured pruning with fine-grained sparsity and quantization presents an intriguing opportunity to reduce inference-time latency while upholding accuracy standards. By making sparsity-aware methods like SD$^2$ more accessible across diverse applications through these advancements, the benefits of efficient LLM inference can be extended suitably to a broader audience.