# Federated Learning of Deep Networks using Model Averaging

**Authors:** H. Brendan McMahan, Eider Moore, Daniel Ramage, Blaise Agüera y Arcas  
**arXiv:** [1602.05629](https://arxiv.org/abs/1602.05629)  
**Date:** 2016-02-17  
**Summary page:** https://www.summarizepaper.com/en/paper/1602.05629v1/

## Key points

- Federated learning is a decentralized approach to training machine learning models on mobile devices
- Modern mobile devices have access to privacy-sensitive and large quantities of data that can improve user experience
- Conventional approaches for training models require logging data in a centralized data center, which is challenging for privacy-sensitive or large datasets
- Federated learning allows the training data to remain distributed on the mobile devices themselves, with a shared model learned by aggregating locally-computed updates from each device
- This approach enables high-quality models to be trained in relatively few rounds of communication, addressing a key constraint in federated learning
- The paper presents a practical method for federated learning of deep networks that is robust to unbalanced and non-IID data distributions commonly found on mobile devices
- Parameter averaging over updates from multiple clients produces surprisingly good results, even when optimizing non-convex loss functions
- The proposed method significantly reduces communication needed to train an LSTM language model by two orders of magnitude
- Federated learning offers an alternative approach for training machine learning models on mobile devices while addressing privacy concerns and large datasets, with promising results in terms of model quality and communication efficiency.

## Summary

The paper titled "Federated Learning of Deep Networks using Model Averaging" discusses the concept of federated learning, which is a decentralized approach to training machine learning models on mobile devices. The authors highlight that modern mobile devices have access to a wealth of data that can be used to improve user experience, such as language models for speech recognition and text entry, and image models for photo selection. However, this data is often privacy-sensitive or large in quantity, making it challenging to train models using conventional approaches that require logging the data in a centralized data center. To address these challenges, the authors propose federated learning, where the training data remains distributed on the mobile devices themselves. The shared model is learned by aggregating locally-computed updates from each device. This approach allows for high-quality models to be trained in relatively few rounds of communication, which is a key constraint in federated learning. The authors present a practical method for federated learning of deep networks that proves robust to unbalanced and non-IID (independent and identically distributed) data distributions commonly found on mobile devices. Despite optimizing non-convex loss functions, they find that parameter averaging over updates from multiple clients produces surprisingly good results. For instance, they demonstrate a significant reduction in communication needed to train an LSTM language model by two orders of magnitude. Overall, this paper provides valuable insights into the potential of federated learning as an alternative approach for training machine learning models on mobile devices while addressing privacy concerns and large datasets. The proposed method offers promising results in terms of model quality and communication efficiency.