# FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

**Authors:** Zhenheng Tang, Yuxin Wang, Xin He, Longteng Zhang, Xinglin Pan, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Bingsheng He, Xiaowen Chu  
**arXiv:** [2309.01172](https://arxiv.org/abs/2309.01172)  
**Date:** 2023-09-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.01172v1/

## Key points

- Rapid growth of memory and computation requirements of large language models (LLMs)
- Challenges for individuals without access to high-end GPUs
- Consumer-level GPUs often overlooked in LLM due to weaker computing performance, smaller storage capacity, and lower communication bandwidth
- Proposal of a decentralized system called FusionAI to leverage consumer-level GPUs for pre-training, inference, and fine-tuning of LLMs while ensuring privacy protection
- Key components of FusionAI include a broker with a backup pool for dynamic join and quit of computing providers, task scheduling based on hardware performance, and abstracting machine learning procedures into directed acyclic graphs (DAGs)
- DAG approach enables model and task universality and provides compatibility across different devices and deep learning frameworks
- Performance analysis shows that 50 RTX 3080 GPUs can achieve comparable throughputs to 4 H100 GPUs at a significantly lower cost
- FusionAI unlocks the potential of consumer-level GPUs for training and deploying LLMs in a decentralized manner while addressing privacy concerns
- Proposed system design effectively tackles challenges related to resource limitations and device heterogeneity

## Summary

The rapid growth of memory and computation requirements of large language models (LLMs) has posed challenges for individuals without access to high-end GPUs. While consumer-level GPUs constitute a larger market share, they are often overlooked in LLM due to their weaker computing performance, smaller storage capacity, and lower communication bandwidth. To address these issues, this paper proposes a decentralized system called FusionAI that leverages the untapped potential of consumer-level GPUs for pre-training, inference, and fine-tuning of LLMs while ensuring privacy protection. The system design incorporates several key components such as a broker with a backup pool to facilitate dynamic join and quit of computing providers and task scheduling taking into account hardware performance to improve overall system efficiency. Additionally, the paper introduces the concept of abstracting machine learning procedures into directed acyclic graphs (DAGs). This approach enables model and task universality by providing a flexible framework for representing various ML tasks. Furthermore, intermediate representation and execution planes are abstracted to ensure compatibility across different devices and deep learning frameworks. The authors conducted performance analysis to evaluate the effectiveness of FusionAI which showed that 50 RTX 3080 GPUs can achieve throughputs comparable to those of 4 H100 GPUs at a significantly lower cost. Overall, FusionAI presents an innovative solution that unlocks the potential of consumer-level GPUs for training and deploying LLMs in a decentralized manner while addressing privacy concerns. The proposed system design effectively tackles challenges related to resource limitations and device heterogeneity, making it a promising approach for democratizing access to large-scale language models.