# On-Device Neural Net Inference with Mobile GPUs

**Authors:** Juhyun Lee, Nikolay Chirkov, Ekaterina Ignasheva, Yury Pisarchyk, Mogan Shieh, Fabio Riccardi, Raman Sarokin, Andrei Kulik, Matthias Grundmann  
**arXiv:** [1907.01989](https://arxiv.org/abs/1907.01989)  
**Date:** 2019-07-03  
**Summary page:** https://www.summarizepaper.com/en/paper/1907.01989v1/

## Key points

- Machine learning models on mobile phones for on-device inference are increasingly popular due to lower latency and increased privacy.
- Hardware accelerators are being used to overcome the challenges of running compute-intensive tasks solely on the mobile CPU.
- Neural processing units in high-end phones account for only a small fraction of hand-held devices.
- The paper titled "On-Device Neural Net Inference with Mobile GPUs" presents an approach to leverage the mobile GPU, which is found in virtually every phone, to run inference of deep neural networks in real-time for both Android and iOS devices.
- The authors' state-of-the-art mobile GPU inference engine is integrated into the open-source project TensorFlow Lite and publicly available at https://tensorflow.org/lite.
- Leveraging the mobile GPU's parallel processing capabilities can achieve faster inference times while maintaining low power consumption.
- Most phones already have a GPU built-in, so there is no need for additional hardware costs or upgrades.

## Summary

The use of machine learning models on mobile phones for on-device inference is becoming increasingly popular due to its lower latency and increased privacy. To overcome the challenges of running such a compute-intensive task solely on the mobile CPU, app developers and researchers have started exploiting hardware accelerators. While device manufacturers are adding neural processing units into high-end phones for on-device inference, they account for only a small fraction of hand-held devices. In this paper titled "On-Device Neural Net Inference with Mobile GPUs," Juhyun Lee and colleagues present their approach to leverage the mobile GPU, which is a ubiquitous hardware accelerator found in virtually every phone, to run inference of deep neural networks in real-time for both Android and iOS devices. The authors describe their architecture and discuss how to design networks that are mobile GPU-friendly. Their state-of-the-art mobile GPU inference engine is integrated into the open-source project TensorFlow Lite and publicly available at https://tensorflow.org/lite. The authors' approach offers several advantages over using the mobile CPU alone or specialized hardware accelerators like neural processing units. By leveraging the mobile GPU's parallel processing capabilities, they can achieve faster inference times while maintaining low power consumption. Additionally, since most phones already have a GPU built in, there is no need for additional hardware costs or upgrades. Overall, this paper presents an innovative solution to overcome the challenges of running machine learning models on mobile devices by leveraging existing hardware resources efficiently. This approach has significant implications for various applications that require real-time inferencing on handheld devices while maintaining privacy and reducing latency.