# Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices

**Authors:** Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, Gennady Pekhimenko  
**arXiv:** [2103.03239](https://arxiv.org/abs/2103.03239)  
**Date:** 2021-03-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.03239v4/

## Key points

- Authors address the challenge of training deep neural networks on large datasets using multiple compute nodes
- Distributed training with specialized message-passing protocols like Ring All-Reduce accelerates the process by leveraging hundreds of computers
- Real-world applications often operate on unreliable devices with unstable network bandwidth, limiting scalability of existing protocols
- Proposal of Moshpit All-Reduce, an iterative averaging protocol designed to exponentially converge to the global average, showcasing speedups in experiments
- Offers strong theoretical guarantees for distributed optimization and enables decentralized training on heterogeneous unreliable devices without traditional methods

## Summary

, , , , 

In their paper titled "Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices," authors Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko address the challenge of training deep neural networks on large datasets using multiple compute nodes. Distributed training is a promising approach that utilizes specialized message-passing protocols like Ring All-Reduce to significantly accelerate this process by leveraging hundreds of computers. However, the scalability of these protocols is limited by the requirement for reliable high-speed networking only available in dedicated clusters. Real-world applications such as federated learning and cloud-based distributed training often operate on unreliable devices with unstable network bandwidth. As a result, these applications are typically constrained to using parameter servers or gossip-based averaging protocols due to the lack of robust communication infrastructure. To overcome this limitation, the authors propose Moshpit All-Reduce - an iterative averaging protocol designed to exponentially converge to the global average. The efficiency of Moshpit All-Reduce is demonstrated through experiments showcasing a 1.3x speedup for ResNet-50 training on ImageNet compared to competitive gossip-based strategies. Additionally, when training ALBERT-large from scratch using preemptible compute nodes, a 1.5x speedup is achieved. The proposed protocol offers strong theoretical guarantees for distributed optimization and enables decentralized training on heterogeneous unreliable devices without relying on traditional parameter servers or gossip-based methods. Accepted at the Conference on Neural Information Processing Systems (NeurIPS) 2021, this work provides a novel solution for improving communication efficiency in distributed deep learning scenarios where reliable high-speed networking is not readily available. The code for Moshpit SGD is openly accessible on GitHub for further exploration and implementation in practical settings.