# (Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court

**Authors:** Angela Jin, Niloufar Salehi  
**arXiv:** [2403.13004](https://arxiv.org/abs/2403.13004)  
**Date:** 2024-03-13  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2403.13004v1/

## Key points

- Challenges faced by public defenders in scrutinizing AI systems in the U.S. criminal legal system
- Difficulties encountered by public defenders in navigating the development and use of computational forensic software (CFS) tools
- Recommendations provided to address challenges, considering technical, social, and institutional contexts
- Importance of understanding downstream stakeholders' needs for performance information and interpretation of it
- Methodology involving participant recruitment strategies using convenience and purposive sampling methods
- Acknowledgment of power imbalances within the criminal legal system impacting low-income and marginalized communities
- Recognition of the need for engagement with criminal defendants and broader communities served by public defenders to understand impacts of AI systems

## Summary

In this paper, we focus on the challenges faced by public defenders in scrutinizing AI systems in the U.S. criminal legal system. Our study involved interviews with 17 individuals from the public defense community, including public defenders, technologists, and legal experts. Through their insights and experiences, we explore the reliability of computational forensic software (CFS) tools such as facial recognition, gunshot detection, and probabilistic genotyping. Our findings reveal that public defenders encounter difficulties in navigating how CFS is developed and used, overcoming non-critical perceptions of CFS by judges and jurors, and accessing expertise on CFS technology. To address these challenges, we provide recommendations that consider the technical, social, and institutional context to support contestability in practice. Additionally, our study references related research on trust in AI models based on stated and observed accuracy. We also examine communication of uncertainty information to decision-makers and jurors' perceptions of error rates associated with forensic evidence. These studies highlight the importance of understanding downstream stakeholders' needs for performance information and how they interpret it. Our methodology section details participant recruitment strategies using convenience and purposive sampling methods to ensure diversity among participants from different regions in the U.S. Interviews were conducted over video calls or phone calls between December 2022 and August 2023. Recordings were transcribed and anonymized for analysis. We acknowledge the power imbalances within the criminal legal system that disproportionately impact low-income and marginalized communities. While our focus is on public defenders' perspectives, we recognize the need for engagement with criminal defendants and broader communities served by public defenders to provide a comprehensive understanding of the impacts of AI systems. Overall, our study aims to shed light on the challenges faced by public defenders in scrutinizing AI systems in high-stakes settings. We propose recommendations to enhance contestability through a nuanced understanding of technical, social, and institutional factors.