# Insights into resource utilization of code small language models serving with runtime engines and execution providers

**Authors:** Francisco Durán, Matias Martinez, Patricia Lago, Silverio Martínez-Fernández  
**arXiv:** [2412.15441](https://arxiv.org/abs/2412.15441)  
**Date:** 2024-12-19  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2412.15441v2/

## Key points

- Rapidly evolving landscape of language models, particularly in code generation
- Pressing need to address substantial computational resources required by these models
- Concerns regarding energy consumption and environmental impact
- Optimizing resource utilization of language model inference is paramount, with Small Language Models (SLMs) emerging as a promising solution
- Impact of deep learning serving configurations on resource utilization within the context of code generation SLMs
- Configurations defined as combinations of runtime engines and execution providers play a crucial role in determining energy consumption, execution time, and computing-resource utilization
- Significant disparities observed across various serving configurations; CUDA outperformed CPU-based providers in terms of energy consumption and execution time
- TORCH paired with CUDA exhibited exceptional energy efficiency, yielding impressive energy savings ranging from 37.99% up to 89.16%
- ONNX coupled with CPU demonstrated notable energy savings ranging from 8.98% up to 72.04% within CPU-based setups
- Choice of serving configuration significantly influences resource utilization metrics such as energy consumption and execution time
- Recommendations for software engineers: leverage TORCH paired with CUDA or ONNX with CPU for enhanced resource utilization efficiency when working with code generation SLMs

## Summary

, , , , 

In the rapidly evolving landscape of language models, particularly in the realm of code generation, there is a pressing need to address the substantial computational resources required by these models. This necessity has sparked concerns regarding energy consumption and environmental impact. In response to this challenge, optimizing the resource utilization of language model inference has become paramount, with Small Language Models (SLMs) emerging as a promising solution to mitigate resource demands. The focus of this study was to delve into the impact of deep learning serving configurations on resource utilization within the context of code generation SLMs. These configurations are defined as combinations of runtime engines and execution providers, which play a crucial role in determining energy consumption, execution time, and computing-resource utilization. The perspective adopted for this analysis was that of software engineers engaged in conducting inference tasks using code generation SLMs. To conduct a comprehensive investigation, a technology-oriented, multi-stage experimental pipeline was implemented utilizing twelve distinct code generation SLMs. Through this rigorous approach, significant disparities were observed across various serving configurations. Notably, configurations employing CUDA as the execution provider consistently outperformed those utilizing CPU-based providers in terms of both energy consumption and execution time. Amongst the myriad configurations explored, it was found that TORCH paired with CUDA exhibited exceptional energy efficiency, yielding impressive energy savings ranging from 37.99% up to 89.16% when compared to alternative serving configurations. Furthermore, configurations featuring optimized runtime engines such as ONNX coupled with CPU execution providers demonstrated notable energy savings ranging from 8.98% up to 72.04% within CPU-based setups. Moreover, TORCH combined with CUDA showcased efficient computing-resource utilization across various experiments conducted during this study. It became evident that the choice of serving configuration significantly influences resource utilization metrics such as energy consumption and execution time. While acknowledging the need for further research in this domain, based on our findings we recommend leveraging TORCH paired with CUDA or ONNX with CPU for software engineers seeking to enhance resource utilization efficiency when working with code generation SLMs. By adopting these optimized configurations, practitioners can potentially achieve substantial improvements in energy efficiency and overall performance while conducting inference tasks within their respective projects.