# EdgeShard: Efficient LLM Inference via Collaborative Edge Computing

**Authors:** Mingjin Zhang, Jiannong Cao, Xiaoming Shen, Zeyang Cui  
**arXiv:** [2405.14371](https://arxiv.org/abs/2405.14371)  
**Date:** 2024-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.14371v1/

## Key points

- Authors address challenges faced by large language models (LLMs) relying on cloud computing:
- Prolonged latency
- High bandwidth costs
- Privacy concerns
- Proposed solution: EdgeShard leveraging edge computing for LLM deployment closer to data sources:
- Partitioning model into shards deployed on distributed devices
- Reduces latency by up to 50% and improves throughput by 2x compared to baseline methods
- Existing methods may result in accuracy loss or unstable network connections
- EdgeShard offers a more effective solution for addressing challenges
- Formulate adaptive joint device selection and model partition problem
- Design efficient dynamic programming algorithm for optimization

## Summary

, , , , 

In their paper titled "EdgeShard: Efficient LLM Inference via Collaborative Edge Computing," authors Mingjin Zhang, Jiannong Cao, Xiaoming Shen, and Zeyang Cui address the challenges faced by large language models (LLMs) that heavily rely on cloud computing. These challenges include prolonged latency, high bandwidth costs, and privacy concerns. To mitigate these issues, the authors propose leveraging edge computing to deploy LLMs on edge devices closer to data sources. Their proposed approach, called EdgeShard, aims to improve the efficiency of LLM inference through collaborative edge computing. This is achieved by partitioning the model into shards and deploying them on distributed devices. By doing so, they are able to reduce latency by up to 50% and improve throughput by 2x compared to baseline methods. While some existing approaches attempt to reduce model size through quantization or utilize cloud-edge collaboration, they often result in accuracy loss or suffer from unstable network connections. In contrast, the authors' approach offers a more effective solution for addressing these challenges. To optimize inference latency and throughput, the authors formulate an adaptive joint device selection and model partition problem. They design an efficient dynamic programming algorithm to achieve this optimization. Overall, this work contributes to advancing the field of natural language processing by offering a novel solution for enhancing the efficiency of LLM inference through collaborative edge computing. It addresses key challenges faced by LLMs and presents a promising approach that can significantly improve performance while also addressing privacy concerns associated with cloud computing.