# Quantitative Toolchain Assurance

**Authors:** Dennis Volpano, Drew Malzahn, Andrew Pareles, Mark Thober  
**arXiv:** [2308.16275](https://arxiv.org/abs/2308.16275)  
**Date:** 2023-08-30  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2308.16275v1/

## Key points

- The software bill of materials (SBOM) concept aims to provide more information about a software build, including copyrights, dependencies, and security references.
- SBOM lacks visibility into the process for building a package.
- Supply-chain Levels for Software Artifacts (SLSA) focuses on the quality of the build process but lacks quantitative assessment.
- A new form of assurance case called process reduction is introduced to fill this gap.
- Process reduction allows for quantitative measurement of the strength of a toolchain by quantifying the strength of the reduction.
- Metrics such as input space coverage, code coverage, and kill ratios should be represented as Beta distributions within the framework.
- Test suite effectiveness can be represented using Beta probability density functions (PDFs).
- Correlation between metrics and finding faults should have its own Beta PDF, with Bayesian updates over time to learn how well the metric correlates with finding faults in a system.
- Attacks on software supply chains highlight the need for reliable and safe systems.
- The proposed approach provides quantitative evidence that falls between informal evidence like SBOMs and more rigorous evidence like proof-carrying code (PCC).
- The introduction of a quantitative assurance case using process reduction measures toolchain strength and assesses its effectiveness in finding faults.

## Summary

The software bill of materials (SBOM) concept aims to provide more information about a software build, including copyrights, dependencies, and security references. However, SBOM lacks visibility into the process for building a package. Efforts such as Supply-chain Levels for Software Artifacts (SLSA) attempt to address this issue by focusing on the quality of the build process. However, these efforts are purely qualitative and lack quantitative assessment of that quality. To fill this gap, a new form of assurance case called process reduction is introduced. This assurance case is quantitative and can measure the strength of a toolchain by quantifying the strength of the reduction. By structuring the assurance case as a process reduction, it becomes possible to assess the effectiveness of a toolchain in terms of finding faults. In addition to metrics such as input space coverage, code coverage, or kill ratios used to evaluate software test suites, these metrics should be represented as Beta distributions within the framework. For example, with mutation testing, successes and failures of a test suite can be represented by a Beta probability density function (PDF), where success is determined by whether the suite causes a mutation in the program under test to exhibit some observable difference in behavior when executed. The effectiveness of a test suite should also be represented within a toolchain since metrics may not always correlate well with finding faults. Correlation should have its own Beta PDF, and Bayesian updates over time can be performed to learn how well the metric correlates with finding faults in a given system. Overall, attacks on software supply chains have highlighted the need for reliable and safe systems. While informal evidence like SBOMs and qualitative efforts like SLSA provide some level of assurance at one end of an evidence spectrum, more rigorous evidence such as proof-carrying code (PCC) exists at the opposite end. The proposed approach aims to provide quantitative evidence for software that falls somewhere between these two endpoints. In conclusion, the introduction of a quantitative assurance case using process reduction provides a means to measure the strength of a toolchain and assess its effectiveness in finding faults. This approach bridges the gap between qualitative efforts and more rigorous evidence, offering a valuable contribution to ensuring reliable and safe software systems.