# Measuring the Energy Consumption and Efficiency of Deep Neural Networks: An Empirical Analysis and Design Recommendations

**Authors:** Charles Edison Tripp, Jordan Perr-Sauer, Jamil Gafur, Amabarish Nag, Avi Purkayastha, Sagi Zisman, Erik A. Bensen  
**arXiv:** [2403.08151](https://arxiv.org/abs/2403.08151)  
**Date:** 2024-03-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.08151v1/

## Key points

- Study focuses on energy consumption of large-scale neural networks, known as the "Red-AI" trend
- BUTTER-E dataset introduced, covering data from 63,527 experimental runs across various configurations
- Non-linear relationship found between energy efficiency and network design challenges traditional thinking
- Importance of algorithm development considering cache effects for designing energy-efficient neural networks
- Recommendations for optimizing software and hardware for more efficient AI task execution
- Concrete action items outlined based on study findings to improve energy efficiency without compromising performance

## Summary

In the face of the escalating energy consumption of large-scale neural networks, commonly referred to as the "Red-AI" trend, this study delves into the actual energy usage during training of various fully connected neural network architectures. The research introduces the BUTTER-E dataset, an extension of the BUTTER Empirical Deep Learning dataset, which encompasses data from 63,527 individual experimental runs across 30,582 distinct configurations. These configurations span 13 datasets, 20 different sizes in terms of trainable parameters (NTPs), 8 network shapes, and 14 depths on both CPU and GPU hardware measured using node-level watt-meters. The  sheds light on the intricate relationship between , , and , while emphasizing the impact of . A straightforward yet effective energy model is proposed in this study that takes into account network size, computing processes, and memory hierarchy. Surprisingly, a non-linear relationship between energy efficiency and network design is uncovered, challenging the notion that reducing parameters or FLOPs is always the best approach for achieving greater energy efficiency. The study underscores the necessity for algorithm development that considers cache effects and suggests a holistic approach to designing energy-efficient neural networks by integrating considerations for software algorithms and hardware design. As AI models continue to grow in complexity and energy costs soar, it becomes imperative to address these issues head-on through empirical measurement and analysis of algorithmic energy costs. Moving forward, further studies are recommended to explore deep learning architectures such as large language models (LLMs), convolutional neural networks (CNNs), and graph neural networks (GNNs) to optimize both software and hardware for more efficient execution of AI tasks. Concrete action items are outlined based on the findings of this study including considerations for network sizing relative to system caches, avoiding wide layers with large input sizes that may lead to inefficient cache utilization, developing cache-aware deep learning approaches, distributing working sets efficiently among processing units to reduce idle time, advocating for larger caches in hardware design, and exploring methods for distributing parameter sets among multiple computing units. By integrating insights from this study into future algorithmic and hardware designs, there is potential to pave the way for more  that do not compromise performance. This work represents a crucial step towards aligning rapid advancements in AI capabilities with efforts to mitigate escalating energy consumption associated with computing tasks.