# Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding

**Authors:** Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, Yu Wang  
**arXiv:** [2307.15337](https://arxiv.org/abs/2307.15337)  
**Date:** 2023-07-28  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2307.15337v2/

## Key points

- Work addresses high generation latency in large language models (LLMs)
- Proposes a new approach called Skeleton-of-Thought (SoT)
- SoT guides LLMs to generate the skeleton of an answer first
- Uses parallel API calls or batched decoding to complete each skeleton point simultaneously
- Reduces generation latency across 12 LLMs
- Potential to improve answer quality for various question categories
- Represents an initial attempt at data-centric optimization for inference efficiency
- Highlights the possibility of pushing LLMs to think more like humans
- Can lead to better answer quality
- Findings underscore the importance of considering alternative approaches to optimize inference efficiency in large language models.

## Summary

This work, titled "Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding," addresses the issue of high generation latency in large language models (LLMs). The authors propose a new approach called Skeleton-of-Thought (SoT), which guides LLMs to first generate the skeleton of an answer and then uses parallel API calls or batched decoding to complete the contents of each skeleton point simultaneously. This novel approach not only significantly reduces generation latency across 12 LLMs but also has the potential to improve answer quality for various question categories. The authors emphasize that SoT represents an initial attempt at data-centric optimization for inference efficiency. It highlights the possibility of pushing LLMs to think more like humans, which can lead to better answer quality. The study's findings underscore the importance of considering alternative approaches to optimize inference efficiency in large language models.