# Training LLMs with Fault Tolerant HSDP on 100,000 GPUs

**Authors:** Omkar Salpekar, Rohan Varma, Kenny Yu, Vladimir Ivanov, Yang Wang, Ahmed Sharif, Min Si, Shawn Xu, Feng Tian, Shengbao Zheng, Tristan Rice, Ankush Garg, Shangfu Peng, Shreyas Siravara, Wenyin Fu, Rodrigo de Castro, Adithya Gangidi, Andrey Obraztsov, Sharan Narang, Sergey Edunov  
**arXiv:** [2602.00277](https://arxiv.org/abs/2602.00277)  
**Date:** 2026-01-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2602.00277v1/

## Key points

- Synchronous training in large-scale systems can be inefficient with O(100K) GPUs due to frequent failures and extended recovery times
- Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP) introduced as a groundbreaking training paradigm for fault tolerance
- FT-HSDP uses data parallel replicas for fault tolerance, allowing the training process to continue even if one replica fails
- Techniques like Fault Tolerant All Reduce (FTAR) protocol and non-blocking catch-up protocol are implemented in FT-HSDP for efficient recovery and seamless rejoining of replicas
- FT-HSDP reduces stall time during failure recovery from 10 minutes to 3 minutes, increasing effective training time from 44% to 80%
- Extensive testing confirms that FT-HSDP's asynchronous recovery process does not compromise model accuracy

## Summary

, , , , 

In the realm of large-scale training systems, synchronous training has been the conventional approach, necessitating all GPUs to be operational simultaneously. However, when dealing with a vast number of GPUs in the order of O(100K), this synchronous training method often proves to be inefficient due to frequent failures and extended recovery times. To combat this challenge, a groundbreaking training paradigm known as Fault Tolerant Hybrid-Shared Data Parallelism (FT-HSDP) has been introduced. FT-HSDP operates by utilizing data parallel replicas as units of fault tolerance. In the event of failures, only a single data-parallel replica containing the faulty GPU or server is temporarily taken offline and restarted, while the other replicas continue with the training process. To implement this innovative concept at scale, FT-HSDP incorporates various techniques such as the introduction of a Fault Tolerant All Reduce (FTAR) protocol for gradient exchange across data parallel replicas. This protocol leverages the CPU to manage intricate control logic tasks such as dynamically adding or removing participants, while relying on GPU for optimal data transfer performance. Additionally, FT-HSDP implements a non-blocking catch-up protocol that enables a recovering replica to seamlessly rejoin training with minimal interruption. Compared to fully synchronous training on O(100K) GPUs, FT-HSDP demonstrates significant improvements by reducing stall time during failure recovery from 10 minutes to just 3 minutes. This enhancement results in an increase in effective training time from 44% to an impressive 80%. Furthermore, extensive testing confirms that FT-HSDP's asynchronous recovery process does not compromise the accuracy of the resulting model. The research conducted by Omkar Salpekar, Rohan Varma, Kenny Yu, Vladimir Ivanov, Yang Wang, Ahmed Sharif, Min Si, Shawn Xu, Feng Tian, Shengbao Zheng and their team sheds light on how FT-HSDP revolutionizes large-scale training systems by enhancing fault tolerance and overall efficiency without sacrificing model accuracy.