# Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization

**Authors:** Xinyuan Zhang, Jiang Liu, Zehui Xiong, Yudong Huang, Gaochang Xie, Ran Zhang  
**arXiv:** [2405.07140](https://arxiv.org/abs/2405.07140)  
**Date:** 2024-05-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.07140v1/

## Key points

- Generative Artificial Intelligence (GAI) and Large Language Models (LLMs) revolutionize content creation
- LLMs are resource-intensive, often requiring cloud hosting
- Edge intelligence traditionally addresses challenges of real-time AI computation on edge devices
- Existing research focuses on conventional AI models, leaving a gap in addressing LLM inference characteristics
- Researchers introduce an optimized model for LLM inference tailored for edge devices
- Approach maximizes inference throughput through batching techniques and model quantization
- Optimization process considers edge resource limitations and user requirements for latency and accuracy
- Researchers propose an optimal Depth-First Tree-Searching algorithm with online tree-pruning (DFTSP) to tackle complexity efficiently
- Simulation results show DFTSP outperforms other benchmarks in terms of throughput and reduces time complexity by over 45%
- Zhang et al.'s work enhances efficiency in large language model inference through innovative techniques

## Summary

Generative Artificial Intelligence (GAI) has revolutionized content creation with its remarkable abilities, and Large Language Models (LLMs) are leading the charge in this domain. However, the resource-intensive nature of LLMs often necessitates cloud hosting, which brings about concerns related to privacy, latency, and usage restrictions. While edge intelligence has traditionally been employed to address these challenges by enabling real-time AI computation on edge devices close to data sources, existing research has primarily focused on conventional AI models, leaving a gap in addressing the unique characteristics of LLM inference. The team introduces an  specifically tailored for . By implementing batching techniques and model quantization on resource-constrained edge devices, they develop an inference model designed for . The primary objective of their approach is to maximize inference throughput through efficient batch scheduling and optimal allocation of communication and computation resources. This optimization process takes into account various factors such as edge resource limitations and diverse user requirements concerning latency and accuracy. To tackle the inherent complexity of this NP-hard problem, the researchers introduce an optimal Depth-First Tree-Searching algorithm with online tree-pruning (DFTSP), which operates within a feasible time complexity. Simulation results demonstrate that DFTSP outperforms other batching benchmarks in terms of throughput across different user settings and quantization techniques. Additionally, it significantly reduces time complexity by over 45% compared to brute-force searching methods. This refined summary showcases the groundbreaking work conducted by Zhang et al., highlighting their contributions towards enhancing efficiency in large language model inference through .