# Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

**Authors:** Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu  
**arXiv:** [2405.19888](https://arxiv.org/abs/2405.19888)  
**Date:** 2024-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.19888v1/

## Key points

- The rise of large language models (LLMs) has revolutionized artificial intelligence, enabling the development of LLM-based applications known as AI agents or co-pilots.
- Existing public LLM services offer only a simplified request-level API, leading to sub-optimal performance due to a lack of application-level information.
- Parrot introduces Semantic Variables as a novel approach to enhance the end-to-end experience of LLM-based applications by conveying application-level knowledge to public LLM services.
- Semantic Variables enable a more natural way to program LLM applications by annotating input/output variables in prompts and establishing data pipelines between multiple LLM requests.
- Parrot allows public LLM services to conduct data flow analysis and uncover correlations across multiple requests, unlocking new optimization opportunities for improving overall performance.
- Extensive evaluations show that Parrot can achieve significant enhancements in popular use cases of LLM applications, with improvements reaching up to an order-of-magnitude.
- Experimental analyses demonstrate how Parrot enhances chain summarization by reducing communication overhead and network latency associated with client interactions compared to baseline approaches like vLLM and HuggingFace.
- Parrot's innovative approach addresses limitations of current public LLM services and paves the way for more efficient utilization of large language models in diverse application scenarios, driving advancements in performance and usability.

## Summary

The rise of large language models (LLMs) has revolutionized the field of artificial intelligence. These powerful models have enabled the development of LLM-based applications, also known as AI agents or co-pilots. By combining traditional software with LLMs, these applications can perform complex tasks through natural language prompts. However, existing public LLM services only offer a simplified request-level API. This lack of application-level information leads to sub-optimal performance. In response to this limitation, Parrot introduces a novel approach focused on enhancing the end-to-end experience of LLM-based applications. Central to Parrot is the concept of Semantic Variables. These serve as a unified abstraction to convey application-level knowledge to public LLM services. By annotating input/output variables in prompts and establishing data pipelines between multiple LLM requests, Semantic Variables enable a more natural way to program LLM applications. Through the utilization of Semantic Variables, Parrot allows public LLM services to conduct conventional data flow analysis and uncover correlations across multiple requests. This correlation unlocks new optimization opportunities for improving overall performance. Extensive evaluations demonstrate that Parrot can achieve significant enhancements in popular use cases of LLM applications. In fact, improvements can reach up to an order-of-magnitude. Additionally, experimental analyses showcase how Parrot enhances chain summarization by reducing communication overhead and network latency associated with client interactions. By optimizing dependent LLM requests and mitigating queuing delays for background requests, Parrot significantly reduces end-to-end latency compared to baseline approaches such as vLLM and HuggingFace. In conclusion, Parrot's innovative approach not only addresses the limitations of current public LLM services but also paves the way for more efficient and effective utilization of large language models in diverse application scenarios. As large language models continue to play a crucial role in shaping the future of AI-driven technologies, solutions like Parrot are poised to drive advancements in performance and usability across various domains.