# Everything, Everywhere All in One Evaluation: Using Multiverse Analysis to Evaluate the Influence of Model Design Decisions on Algorithmic Fairness

**Authors:** Jan Simson, Florian Pfisterer, Christoph Kern  
**arXiv:** [2308.16681](https://arxiv.org/abs/2308.16681)  
**Date:** 2023-08-31  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2308.16681v1/

## Key points

- Algorithmic decision making (ADM) systems can offer objective decisions and save resources
- ADM systems can also lead to unfair outcomes that discriminate against certain societal groups
- Biases in data can be either mitigated or reinforced throughout the modeling process of ADM systems
- Multiverse analysis is a method proposed for algorithmic fairness, involving turning implicit design decisions into explicit ones and examining their implications for fairness
- Metrics of fairness and performance are computed for each universe created through different decision combinations in multiverse analysis
- Design decisions during the development of a machine learning system can have surprising effects on its fairness
- Multiverse analysis helps detect these effects and gain insights into the variability and robustness of algorithmic fairness
- Transparency in ADM system design and implementation is important to ensure fair outcomes
- Multiverse analysis can facilitate discussions between developers and stakeholders on the design of ADM systems
- Collaboration between different disciplines, such as psychology, is needed to address challenges in algorithmic fairness.

## Summary

In this study, the authors explore the use of algorithmic decision making (ADM) systems and their potential impact on fairness. ADM systems can offer more objective decisions and save resources but can also lead to unfair outcomes that discriminate against certain societal groups. The downstream effects of these systems depend on design decisions made during their development as biases in data can be either mitigated or reinforced throughout the modeling process. To address this issue, the authors propose a method called multiverse analysis for algorithmic fairness. This approach involves turning implicit design decisions into explicit ones and examining their implications for fairness by combining different decisions to create a grid of all possible "universes" of decision combinations. For each universe, metrics of fairness and performance are computed to understand how specific design choices impact fairness. Through a case study focused on predicting public health coverage for vulnerable populations, the authors demonstrate how various design decisions during the development of a machine learning system can have surprising effects on its fairness. By using multiverse analysis they are able to detect these effects and gain insights into the variability and robustness of algorithmic fairness. The results emphasize the importance of explicitly considering design decisions in ADM systems and understanding how they influence fairness outcomes. Additionally, it is argued that multiverse analysis can inform discussions between developers and stakeholders facilitating joint reflections on the ultimate design of ADM systems. This study highlights the need for transparency in ADM system design and implementation to ensure fair outcomes while demonstrating the feasibility of adapting methods from different disciplines such as psychology to address challenges in algorithmic fairness. It is hoped that this interdisciplinary approach will inspire further collaboration and exchange of ideas between research fields.