# Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents

**Authors:** Yueying Li, Jim Dai, Tianyi Peng  
**arXiv:** [2504.07347](https://arxiv.org/abs/2504.07347)  
**Date:** 2025-04-10  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2504.07347v1/

## Key points

- Increasing demand for Large Language Models (LLMs) and AI agents
- Importance of optimizing systems for efficient LLM inference
- Use of mathematical modeling and queuing theory to develop fundamental principles
- Exploration of throughput in LLM inference systems
- 'Work-conserving' scheduling algorithms achieve maximum throughput
- Highlighting 'work-conserving' as a key design principle for system performance enhancement
- Orca and Sarathi-Serve identified as throughput-optimal solutions, caution against FastTransformer and vanilla vLLM due to instability issues
- Benefits of incorporating queuing theory in enhancing LLM inference systems
- Insights into latency optimization, algorithm selection based on different factors
- Token budget choices in Sarathi-Serve impact latency performance, revealing challenges and opportunities for future research
- Influence of token budget sizes on end-to-end latency and prefill time demonstrated through experiments using the CodeLlama-34B model

## Summary

The paper titled "Throughput-Optimal Scheduling Algorithms for LLM Inference and AI Agents" addresses the increasing demand for Large Language Models (LLMs) and AI agents. It emphasizes the importance of optimizing systems for efficient LLM inference and delves into mathematical modeling and queuing theory to develop fundamental principles. The research aims to bridge the gap between queuing theory and LLM system communities by exploring throughput in LLM inference systems. The authors demonstrate that a class of 'work-conserving' scheduling algorithms can achieve maximum throughput for individual requests and AI agent workloads. They highlight 'work-conserving' as a key design principle that enhances system performance. Real-world evaluations showcase Orca and Sarathi-Serve as throughput-optimal solutions while cautioning against FastTransformer and vanilla vLLM due to instability issues. These findings underscore the significant benefits of incorporating queuing theory in enhancing LLM inference systems, calling for further interdisciplinary collaborations. Additionally, the paper introduces insights into latency optimization, acknowledging that while optimal throughput algorithms have been developed, selecting the algorithm with the lowest latency can vary based on different factors. Preliminary analyses on how token budget choices in Sarathi-Serve impact latency performance reveal both challenges and opportunities for future research. Experiments using the CodeLlama-34B model illustrate how token budget sizes influence end-to-end latency and prefill time, highlighting trade-offs involved in latency optimization. Overall, this study contributes valuable insights into maximizing system efficiency through queuing theory principles while also shedding light on considerations for latency optimization in LLM inference systems. These findings emphasize the need for continued interdisciplinary advancements to further enhance these critical technologies.