# Towards Efficient and Reliable LLM Serving: A Real-World Workload Study

**Authors:** Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, Xiaowen Chu  
**arXiv:** [2401.17644](https://arxiv.org/abs/2401.17644)  
**Date:** 2024-01-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.17644v1/

## Key points

- Large language models (LLMs), especially GPT models, have advanced significantly in recent years
- Challenges in broader development due to high operational and deployment costs
- Active research on improving hardware efficiency of LLMs
- Lack of reliable workload data for evaluating LLM serving systems impacts QoS and reliability
- Introduction of the first real-world trace dataset of LLM serving workloads providing insights into user behavior, system performance, and interactions
- Identification of burstiness in workload requests and responses through trace data analysis
- Development of a benchmark suite reflecting workload patterns for performance evaluation and precise scaling
- Uncovering vulnerability of LLM serving systems to short-term burstiness due to GPU memory limitations causing performance degradation
- Importance of understanding workload patterns for optimizing LLM workload management and enabling elastic hardware resource adjustments effectively
- Plan to make dataset and benchmark suite publicly available to encourage further research

## Summary

Large language models (LLMs), particularly Generative Pretrained Transformer (GPT) models, have made significant advancements in the industry in recent years. However, the broader development of these models faces challenges due to high operational and deployment costs. To address this issue, there has been active research focused on improving the hardware efficiency of LLMs. Despite these efforts, the characteristics of real-world LLM workloads are often overlooked in current optimizations of LLM serving systems. In a recent study, researchers identified a lack of reliable workload data for evaluating LLM serving systems, which impacts the quality of service (QoS) and reliability in industrial deployments. To address this gap, the researchers introduced the first real-world trace dataset of LLM serving workloads, providing detailed insights into user behavior, system performance, and LLM interactions. By analyzing this trace data, they highlighted patterns such as burstiness in workload requests and responses. Based on their findings, they developed a benchmark suite that reflects the workload patterns observed in their dataset. This suite enables performance evaluation of LLM serving systems by capturing core workload distributions and allowing for precise scaling to match system sizes. Through their evaluation, they uncovered a previously unrecognized vulnerability of LLM serving systems to short-term burstiness, particularly in common workload scenarios. One key observation from the study was that GPU memory limitations caused by fluctuating burstiness led to significant performance degradation in existing LLM serving systems. Understanding these patterns is crucial for optimizing LLM workload management and enabling elastic hardware resource adjustments to accommodate varying workloads effectively. The researchers plan to make their dataset and benchmark suite publicly available to encourage further research in this area. By shedding light on real-world LLM workload behaviors and developing tools for performance evaluation,this study contributes valuable insights that can help enhance the efficiency and reliability of LLM serving systems in industrial deployments.