# Boosting Performance of Iterative Applications on GPUs: Kernel Batching with CUDA Graphs

**Authors:** Jonah Ekelund, Stefano Markidis, Ivy Peng  
**arXiv:** [2501.09398](https://arxiv.org/abs/2501.09398)  
**Date:** 2025-01-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.09398v1/

## Key points

- Authors Jonah Ekelund, Stefano Markidis, and Ivy Peng focus on accelerating scientific applications on GPUs.
- Introduction of CUDA Graph as a graph-based execution model to optimize GPU performance by consolidating kernel launches.
- Proposal of a performance optimization strategy for iteratively launched kernels through grouping launches into iteration batches and unrolling them into CUDA Graphs.
- Demonstration of an optimal size for iteration batches that yields more than 1.4 times speed-up in the skeleton application.
- Extension of findings to showcase speed-up results in benchmark suites like Hotspot, Hotspot3D, and FDTD Maxwell solver.
- Research highlights significant performance improvements in GPU-accelerated iterative applications through strategic kernel batching with CUDA Graphs.

## Summary

In their paper titled "Boosting Performance of Iterative Applications on GPUs: Kernel Batching with CUDA Graphs," authors Jonah Ekelund, Stefano Markidis, and Ivy Peng delve into the realm of Graphics Processing Units (GPUs) and their role in accelerating scientific applications on heterogeneous systems. With the increasing speed of GPUs, a notable performance bottleneck arises from launching multiple fine-grained kernels. To address this challenge, the authors introduce CUDA Graph, a graph-based execution model that represents operations as nodes and dependencies as edges in a static graph. By consolidating multiple kernel launches into a single graph launch, CUDA Graph aims to optimize performance for GPU-accelerated applications. The paper proposes a novel performance optimization strategy specifically tailored for iteratively launched kernels. By grouping kernel launches into iteration batches and subsequently unrolling these batches into a CUDA Graph, iterative applications can harness the benefits of CUDA Graph for enhanced performance. The authors conduct an in-depth analysis of the performance gains and overhead associated with this approach by designing a skeleton application. This application serves as both a practical example of converting an iterative solver to utilize CUDA Graph and as a basis for deriving a performance model. Through experimentation with the skeleton application, the authors demonstrate that there exists an optimal size for iteration batches when unrolled for a given platform. This optimal batch size remains independent of workload and strikes a balance between the additional overhead incurred from graph creation and the performance enhancement achieved through graph execution. Depending on the workload characteristics, the optimal iteration batch size yields more than 1.4 times speed-up in the skeleton application. Furthermore, the authors extend their findings to showcase similar speed-up results in popular benchmark suites such as Hotspot and Hotspot3D from the Rodinia suite, as well as in a Finite-Difference Time-Domain (FDTD) Maxwell solver. By highlighting the efficacy of their proposed approach across various applications, Ekelund et al. 's research underscores the potential for significant performance improvements in GPU-accelerated iterative applications through strategic kernel batching with CUDA Graphs.