# Photon: Federated LLM Pre-Training

**Authors:** Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, Nicholas D. Lane  
**arXiv:** [2411.02908](https://arxiv.org/abs/2411.02908)  
**Date:** 2024-11-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.02908v1/

## Key points

- Photon is a system designed for federated end-to-end training of large language models (LLMs) in a distributed environment.
- It leverages low-bandwidth methods like federated learning (FL) to enable collaborative training across weakly-connected GPUs, with a focus on pre-training.
- Photon's key strength lies in its robustness to data heterogeneity and its ability to converge twice as fast as previous methods such as DiLoCo.
- The system achieves enhanced efficiency through small client batch sizes and high learning rates enabled by federated averaging's robustness to hyperparameters.
- Photon can train model sizes up to 7B in a federated fashion while achieving better perplexity than centralized pre-training methods, showcasing its value for efficient and scalable LLM training solutions.
- It outperforms baseline distributed training methods in wall-time efficiency by communicating significantly less, resulting in a 35% improvement and reducing model training time with increased compute availability.
- Photon introduces the first complete system for federated LLM training, utilizing cross-silo FL for global-scale training with minimal communication overheads, enabling the training of decoder-only LLMs from scratch while minimizing bandwidth demands.

## Summary

Photon: A Revolutionary System for Federated End-to-End Training of Large Language Models

Photon is an innovative system designed to address the challenges of training large language models (LLMs) in a distributed environment. Leveraging low-bandwidth methods like federated learning (FL), Photon enables collaborative training across weakly-connected GPUs, with a focus on pre-training. This groundbreaking approach allows for the first complete system for federated LLM training, resulting in impressive performance metrics. One key strength of Photon lies in its robustness to data heterogeneity and its ability to converge twice as fast as previous methods such as DiLoCo. This enhanced efficiency is achieved through a unique strategy that combines small client batch sizes with exceptionally high learning rates, made possible by the robustness of federated averaging to hyperparameters. Photon's innovative approach and impressive performance metrics make it a valuable tool for researchers and practitioners seeking efficient and scalable solutions for training large language models in distributed environments. With the ability to train model sizes up to 7B in a federated fashion while achieving even better perplexity than centralized pre-training methods, Photon represents a significant milestone in enabling economical global internet-wide pre-training of LLMs. Notably, Photon outperforms baseline distributed training methods in terms of wall-time efficiency by communicating significantly less—up to 64x-512x less—resulting in a 35% improvement. This showcases the system's capability to reduce model training time with increased compute availability, achieving a similar compute-time trade-off as centralized approaches. In a significant advancement, Photon introduces the first complete system for federated LLM training, utilizing cross-silo FL for global-scale training with minimal communication overheads. This allows for the training of the first federated family of decoder-only LLMs from scratch, showcasing Photon's capability to train large models while minimizing bandwidth demands. Traditionally, data centers have been limited to resources required for scaling large language models due to high-bandwidth demands of distributed training. However, Photon's use of low-bandwidth methods like federated learning enables collaborative training across weakly-connected GPUs in distributed environments, making it a groundbreaking system for end-to-end training of LLMs.