# A Study on the Intersection of GPU Utilization and CNN Inference

**Authors:** Jack Kosaian, Amar Phanishayee  
**arXiv:** [2212.07936](https://arxiv.org/abs/2212.07936)  
**Date:** 2022-12-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.07936v1/

## Key points

- Significant progress in developing neural network architectures for high predictive performance and application-level inference throughput
- Importance of GPU utilization during inference
- High GPU utilization crucial for increasing application-level throughput and ROI
- Analysis of GPU utilization of convolutional neural network (CNN) inference
- Many CNNs have room to enhance their GPU utilization
- Exploration of GPU utilization within a neural architecture search (NAS) search space
- Proposal to use GPU utilization as a metric to accelerate NAS itself
- Designing more efficient networks by considering GPU utilization during architecture search process
- Need to improve inference-time GPU utilization of CNNs
- Knowledge of GPU utilization can benefit applications beyond optimizing resource usage
- Findings hope to inspire future innovation in designing more efficient and GPU-utilization-friendly neural networks

## Summary

In recent years, there has been significant progress in developing neural network architectures that achieve high predictive performance and application-level inference throughput. However, another important metric that is gaining importance is GPU utilization during inference. GPU utilization measures how effectively a deployed neural network utilizes the computational capabilities of the GPU on which it runs. Achieving high GPU utilization is crucial for increasing application-level throughput and ensuring a good return on investment for deploying GPUs. This paper focuses on analyzing the GPU utilization of convolutional neural network (CNN) inference. The authors first survey the GPU utilization of CNNs and identify areas where improvement is needed. They find that many CNNs have room to enhance their GPU utilization. To further investigate this issue, they explore the GPU utilization of networks within a neural architecture search (NAS) search space. The authors also propose using GPU utilization as a metric to potentially accelerate NAS itself. By considering GPU utilization during the architecture search process, researchers can design more efficient networks that make better use of available computational resources. The study highlights the need to improve the inference-time GPU utilization of CNNs and emphasizes that knowledge of GPU utilization can benefit applications beyond just optimizing resource usage. The authors hope that their findings will inspire future innovation in designing more efficient and GPU-utilization-friendly neural networks. Overall, this research sheds light on the intersection between GPU utilization and CNN inference, providing insights into how to improve efficiency and maximize the benefits of deploying GPUs in deep learning applications.