# Estimating Test Performance for AI Medical Devices under Distribution Shift with Conformal Prediction

**Authors:** Charles Lu, Syed Rakin Ahmed, Praveer Singh, Jayashree Kalpathy-Cramer  
**arXiv:** [2207.05796](https://arxiv.org/abs/2207.05796)  
**Date:** 2022-07-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.05796v1/

## Key points

- Development and deployment of AI-based medical devices require thorough evaluation of safety, efficiency, and usability.
- Estimating test performance under distribution shifts is crucial to ensure robustness and trustworthiness in clinical settings.
- Acquiring labeled medical datasets for this purpose is challenging due to regulatory constraints.
- "Black-box" test estimation technique based on conformal prediction predicts test accuracy of an arbitrary black-box model on an unlabeled target domain without modifying the original training process or making any distributional assumptions about the source data.
- Proposed technique outperforms other methods in terms of accuracy estimation while being practical and effective for black-box models.
- Recent works have investigated techniques and frameworks for estimating test performance on unlabeled domain-shifted distributions.
- Standardized evaluation procedures will improve the robustness and trustworthiness of clinical AI tools.

## Summary

The development and deployment of AI-based medical devices require thorough evaluation of their safety, efficiency, and usability. Estimating the test performance of such devices under distribution shifts is crucial to ensure their robustness and trustworthiness in clinical settings. However, acquiring large amounts of labeled medical datasets for this purpose is challenging due to regulatory constraints. Therefore, in this study, the authors propose a "black-box" test estimation technique based on conformal prediction that predicts the test accuracy of an arbitrary black-box model on an unlabeled target domain without modifying the original training process or making any distributional assumptions about the source data. To evaluate their proposed technique, the authors compare it with other methods on three medical imaging datasets (mammography, dermatology, and histopathology) under several clinically relevant types of distribution shift (institution, hardware scanner, atlas, hospital). They find that their method outperforms other techniques in terms of accuracy estimation while being practical and effective for black-box models. The problem of identifying and rectifying performance degradation under new data populations has been extensively studied as distribution shift, out-of-distribution detection, and domain generalization. Recent works have begun to investigate techniques and frameworks for estimating test performance on unlabeled domain-shifted distributions. Deng & Zheng (2020) introduced the notion of predicting performance on an unlabeled test set using feature vectors from models trained under different distribution shifts. Garg et al. (2022) proposed a simpler technique that estimates accuracy on an unlabeled target distribution by selecting a confidence threshold using accuracy on a source dataset. In conclusion, this study contributes to promoting practical and effective estimation techniques for black-box models used in medical device software. The authors hope that these standardized evaluation procedures will improve the robustness and trustworthiness of clinical AI tools. This paper was presented at ICML Workshop on Principles of Distribution Shift (PODS) 2022.