# Rapid Regression Detection in Software Deployments through Sequential Testing

**Authors:** Michael Lindon, Chris Sanden, Vaché Shirikian  
**arXiv:** [2205.14762](https://arxiv.org/abs/2205.14762)  
**Date:** 2022-05-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2205.14762v2/

## Key points

- Continuous deployment revolutionizes software industry by reducing time-to-market
- Improved efficiency and productivity in development process
- Inherent risk of releasing defective changes
- Defects can negatively impact user experience and lead to increased user abandonment
- Quality control gates are essential in software delivery process
- Canary tests help ensure quality control by evaluating new software under production workloads
- Statistical framework for rapidly detecting regressions in software deployments presented
- Framework enables rapid detection of regressions while controlling false detection probability
- Case studies conducted at Netflix demonstrate utility of the approach
- Implementing statistical framework enhances ability to deliver reliable software updates while minimizing risk of defects impacting user experience and abandonment
- Importance of quality control gates highlighted, practical solution presented for detecting regressions in software deployments
- Findings have significant implications for optimizing software delivery processes and maintaining positive user experience for internet companies.

## Summary

The practice of continuous deployment has revolutionized the software industry by allowing companies to reduce time-to-market through faster software deployment. This approach has significantly improved efficiency and productivity in the development process. However, it also comes with the inherent risk of occasionally releasing defective changes. For internet companies, such defects can have a detrimental impact on user experience and lead to increased user abandonment. To mitigate these risks, quality control gates are essential in the software delivery process. These gates help build confidence in the reliability of a release or change and ensure high-quality releases. One common approach to ensure quality control is performing canary tests, which involve evaluating new software under production workloads. The goal is to detect defects as early as possible to minimize exposure and provide immediate feedback to developers. In this paper, the authors present a statistical framework for rapidly detecting regressions in software deployments. The proposed approach is based on sequential tests of stochastic order and equality in distribution. By continuously monitoring canary tests, this framework enables rapid detection of regressions while strictly controlling the false detection probability throughout the process. The authors demonstrate the utility of their approach through two case studies conducted at Netflix. This research contributes valuable insights into improving software deployment practices by effectively detecting regressions and ensuring high-quality releases. By implementing this statistical framework, companies can enhance their ability to deliver reliable software updates while minimizing the risk of introducing defects that could negatively impact user experience and lead to user abandonment. Overall, this study highlights the importance of quality control gates and presents a practical solution for rapidly detecting regressions in software deployments. The findings have significant implications for internet companies seeking to optimize their software delivery processes and maintain a positive user experience.