# Large Language Models for Compiler Optimization

**Authors:** Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Kim Hazelwood, Gabriel Synnaeve, Hugh Leather  
**arXiv:** [2309.07062](https://arxiv.org/abs/2309.07062)  
**Date:** 2023-09-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2309.07062v1/

## Key points

- Large Language Models (LLMs) applied to code optimization
- 7B-parameter transformer model trained from scratch for LLVM assembly optimization
- Model predicts instruction counts before and after optimization, as well as optimized code
- Auxiliary learning tasks enhance model's performance and understanding
- Achieves 3.0% improvement in reducing instruction counts compared to the compiler
- Outperforms two state-of-the-art baselines requiring thousands of compilations
- Strong code reasoning abilities: generates compilable code 91% of the time, emulates compiler output 70% of the time
- Unique focus on optimizing code compared to other LLMs trained on source code for different tasks
- Demonstrates potential of LLMs in improving code performance through automated optimizations

## Summary

This paper explores the novel application of Large Language Models (LLMs) to code optimization. The authors present a 7B-parameter transformer model that is trained from scratch to optimize LLVM assembly for code size. The model takes unoptimized assembly as input and outputs a list of compiler options to best optimize the program. During training, the model is asked to predict instruction counts before and after optimization, as well as the optimized code itself. These auxiliary learning tasks significantly improve the optimization performance of the model and enhance its depth of understanding. The authors evaluate their approach on a large suite of test programs and find that it achieves a 3.0% improvement in reducing instruction counts compared to the compiler. This outperforms two state-of-the-art baselines that require thousands of compilations. Additionally, the model demonstrates strong code reasoning abilities, generating compilable code 91% of the time and perfectly emulating the output of the compiler 70% of the time. While there have been previous LLMs trained on source code for various tasks such as code search, summarization, and documentation generation, this work is unique in its focus on optimizing code. Most LLMs are trained at least partly on code, but this paper specifically targets programming language models for optimization purposes. Overall, this research showcases how Large Language Models can be effectively utilized for compiler optimization and highlights their potential in improving code performance through automated optimizations.