# APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

**Authors:** Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang, Jie Tang, Yuxiao Dong  
**arXiv:** [2401.06761](https://arxiv.org/abs/2401.06761)  
**Date:** 2024-01-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2401.06761v1/

## Key points

- Introduction of parallel auto-regressive generation method to enhance efficiency of large language models (LLMs) in serving scenarios
- Instruction-tuning on general domain data with hierarchical structures allows LLMs to independently plan and perform auto-parallel auto-regressive (APAR) generation
- Up to 2x speed-up achieved, up to 4x when combined with speculative decoding
- APAR reduces key-value cache consumption and attention computation during generation, leading to throughput increase of 20-70% and latency reduction of 20-35%
- Structuring paragraphs in root-and-details format for accurate extraction of hierarchical structures from LLM responses
- Excluding unstructured data like code and math content from structure extraction but using as negative examples during model training
- Experimental setup involves fine-tuning APAR on vicuna-v1.3-{7B,13B} models for Vanilla-APAR, Medusa-APAR, and Batched-APAR implementations
- Training samples drawn from structured and unstructured data sources at a ratio of 1:1; specific batch sizes and learning rates used before training additional medusa heads for improved performance
- Highlighting how APAR enhances LLM generation efficiency by enabling parallel decoding threads while maintaining coherence and accuracy in text prediction across serving scenarios

## Summary

This work introduces a parallel auto-regressive generation method to enhance the efficiency of large language models (LLMs) in various serving scenarios. By instruct-tuning on general domain data with hierarchical structures, LLMs can independently plan their generation process and perform auto-parallel auto-regressive (APAR) generation. This approach results in up to a 2x speed-up and up to 4x when combined with speculative decoding. Furthermore, APAR reduces key-value cache consumption and attention computation during generation, resulting in a throughput increase of 20-70% and a latency reduction of 20-35% compared to existing serving frameworks. To accurately extract hierarchical structures from LLM responses, paragraphs are structured in a root-and-details format where the first sentence serves as the root summarizing the main idea. Unstructured data such as code and math content is excluded from structure extraction but included as negative examples during model training to prevent excessive branching during decoding. The experimental setup involves fine-tuning APAR on vicuna-v1.3-{7B,13B} models to produce APAR-{7B,13B}, which are evaluated using three settings: Vanilla-APAR implemented with transformers, Medusa-APAR implemented with Medusa for speculative decoding, and Batched-APAR implemented with vLLM for high-throughput serving scenarios. During training, samples are drawn from both structured and unstructured data sources at a ratio of 1:1. Models are fine-tuned with specific batch sizes and learning rates before training additional medusa heads for improved performance. Overall, this work highlights how APAR enhances LLM generation efficiency by enabling parallel decoding threads while maintaining coherence and accuracy in text prediction across various serving scenarios.