# Moving Faster and Reducing Risk: Using LLMs in Release Deployment

**Authors:** Rui Abreu, Vijayaraghavan Murali, Peter C Rigby, Chandra Maddila, Weiyan Sun, Jun Ge, Kaavya Chinniah, Audris Mockus, Megh Mehta, Nachiappan Nagappan  
**arXiv:** [2410.06351](https://arxiv.org/abs/2410.06351)  
**Date:** 2024-10-08  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2410.06351v1/

## Key points

- Traditional focus in release engineering: delivering features and bug fixes continuously
- Challenge at scale for release engineering teams to determine which changes should be released
- Development of diff risk score (DRS) models to assess likelihood of severe faults caused by diffs
- Gating risky code changes based on risk thresholds (green, weekend, yellow, red)
- Research approaches explored: logistic regression models, BERT-based models like StarBERT, generative LLMs like iCodeLlama-34B and iDiffLlama-13B
- Performance comparison: generative LLMs show better performance in capturing severe faults compared to regression and BERT-based models
- Providing developers with detailed information on risk scores in Phabricator UI for feedback and actions to reduce risks
- Use of predictors in models to prioritize actionable factors while maintaining developer understanding
- Future potential benefits of exploring large language models for incorporating content-based features into risk assessment

## Summary

, , , , 

In the realm of release engineering, the focus has traditionally been on delivering features and bug fixes to users in a continuous manner. However, as organizations like Meta reach a certain scale, it becomes increasingly challenging for release engineering teams to determine which changes should be released. This responsibility then shifts back to the engineers who are writing and reviewing the code. To tackle this challenge, models of diff risk scores (DRS) have been developed to assess the likelihood of a diff causing a severe fault that impacts end-users. The aim is to build a model that can capture a high percentage of these severe faults by gating only a fraction of incoming code changes. By training the model on historical data of diffs that have led to severe faults in the past, it becomes possible to predict the riskiness of an outgoing diff. Diffs that exceed a certain threshold of risk can then be gated using different levels such as no gating (green), weekend gating (weekend), medium impact on end-users (yellow), and high impact on end-users (red). Various research approaches have been explored, including logistic regression models, BERT-based models like StarBERT, and generative LLMs such as iCodeLlama-34B and iDiffLlama-13B. The baseline regression model captures a significant percentage of severe faults when gating different percentages of risky diffs. While BERT-based models show some improvement over regression models, generative LLMs demonstrate even better performance in capturing severe faults. Furthermore, developers are provided with detailed information in the Phabricator UI regarding the risk score of their diffs, feedback mechanisms for providing input on risk scores, reasons for considering a diff risky, and potential actions to reduce risk scores and successfully land changes. The use of various predictors in these models helps prioritize actionable factors while balancing performance and ease of understanding for developers. Moving forward, exploring large language models offers potential benefits by incorporating content-based features like code changes and test plans into risk assessment. This allows for a more comprehensive understanding of diff risks beyond metadata analysis alone. Overall, this paper delves into the complexities of software development at Meta, evolution in code freeze practices, presentation of gated diff risks to developers through UI elements like risk scores and feedback mechanisms, as well as an in-depth analysis of different risk modeling techniques aimed at improving release engineering processes.