Estimating Test Performance for AI Medical Devices under Distribution Shift with Conformal Prediction

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   Label-free estimation of clinically relevant performance metrics under distribution shifts

   Tim Flühmann, Alceu Bissoto, Trung-Dung Hoang et al. · 2025 · UNSURE@MICCAI · cited 2 times · Not summarized yet

   Abstract

   Performance monitoring is essential for safe clinical deployment of image classification models. However, because ground-truth labels are typically unavailable in the target dataset, direct assessment of real-world model performance is infeasible. State-of-the-art performance estimation methods address this by leveraging confidence scores to estimate the target accuracy. Despite being a promising direction, the established methods mainly estimate the model's accuracy and are rarely evaluated in a clinical domain, where strong class imbalances and dataset shifts are common. Our contributions are twofold: First, we introduce generalisations of existing performance prediction methods that direc…

2. Very close Not relevant

   Automatic Dataset Shift Identification to Support Safe Deployment of Medical Imaging AI

   Mélanie Roschewitz, Raghav Mehta, Charles Jones et al. · 2024 · International Conference on Medical Image Computing and Computer-Assisted Intervention · cited 11 times · Not summarized yet

   Abstract

   Shifts in data distribution can substantially harm the performance of clinical AI models and lead to misdiagnosis. Hence, various methods have been developed to detect the presence of such shifts at deployment time. However, the root causes of dataset shifts are diverse, and the choice of shift mitigation strategies is highly dependent on the precise type of shift encountered at test time. As such, detecting test-time dataset shift is not sufficient: precisely identifying which type of shift has occurred is critical. In this work, we propose the first unsupervised dataset shift identification framework for imaging datasets, effectively distinguishing between prevalence shift (caused by a cha…

3. Very close Not relevant

   Conformal Triage for Medical Imaging AI Deployment

   Anastasios Nikolas Angelopoulos, S. Pomerantz, S. Do et al. · 2024 · medRxiv · preprint, not peer reviewed · cited 15 times · Not summarized yet

   Abstract

   Background: The deployment of black-box AI models in medical imaging presents significant challenges, especially in maintaining reliability across different clinical settings. These challenges are compounded by distribution shifts that can lead to failures in reproducing the accuracy attained during the AI model's original validations. Method: We introduce the conformal triage algorithm, designed to categorize patients into low-risk, high-risk, and uncertain groups within a clinical deployment setting. This method leverages a combination of a black-box AI model and conformal prediction techniques to offer statistical guarantees of predictive power for each group. The high-risk group is guara…

4. Close Not relevant

   Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

   Annie Ulichney, Amanda Coston · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general…

5. Close Not relevant

   Improving Trustworthiness of AI Disease Severity Rating in Medical Imaging with Ordinal Conformal Prediction Sets

   Charles Lu, Anastasios Nikolas Angelopoulos, S. Pomerantz · 2022 · International Conference on Medical Image Computing and Computer-Assisted Intervention · cited 65 times · Not summarized yet

   Abstract

   The regulatory approval and broad clinical deployment of medical AI have been hampered by the perception that deep learning models fail in unpredictable and possibly catastrophic ways. A lack of statistically rigorous uncertainty quantification is a significant factor undermining trust in AI results. Recent developments in distribution-free uncertainty quantification present practical solutions for these issues by providing reliability guarantees for black-box models on arbitrary data distributions as formally valid finite-sample prediction intervals. Our work applies these new uncertainty quantification methods — specifically conformal prediction — to a deep-learning model for grading the severit…

6. Close Not relevant

   Multi-Metric Trustworthiness Evaluation of AI-Assisted Medical Imaging Diagnosis: Integrating Confidence Calibration and Distribution Shift Detection

   Yan-Hua Chen, Jiawen Lai · 2026 · Journal of Global Engineering Review · cited 3 times · Not summarized yet

   Abstract

   The rapid proliferation of artificial intelligence in medical imaging diagnosis has created an urgent need for comprehensive trustworthiness evaluation frameworks. This study proposes a multi-metric evaluation methodology that integrates confidence calibration assessment with distribution shift detection to address the critical challenge of silent failures in deployed AI diagnostic systems. The proposed framework encompasses Expected Calibration Error measurement across demographic subgroups, statistical distance-based distribution shift detection in deep feature spaces, and a novel trustworthiness scorecard that synthesizes prediction confidence, distributional similarity, and historical ac…

7. Close Cited by this paper A method it uses Not relevant

   Detecting and Correcting for Label Shift with Black Box Predictors

   Zachary Chase Lipton, Yu-Xiang Wang, Alex Smola · 2018 · International Conference on Machine Learning · cited 752 times · Not summarized yet

   Abstract

   Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets), cause symptoms (observations), we focus on label shift, where the label marginal $p(y)$ changes but the conditional $p(x|y)$ does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution $p(y)$. BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are inv…

8. Close Not relevant

   Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples

   F. Yilmaz, Reinhard Heckel · 2022 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

   Abstract

   Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes containing the correct class with a user-specified probability based on the classifier's probability estimates. To provide such sets, conformal predictors often estimate a cutoff threshold for the probability estimates based on a calibration set. Conformal predictors guarantee reliability only when the calibration set is from the same distribution as the test set. Therefore, conformal predictors need to be recalibrated for new distributions. However, in practice, labeled data from new distributions is rarely avail…

Prior work it builds on

1. Close Not relevant

   Predicting with Confidence on Unseen Distributions

   Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi et al. · 2021 · IEEE International Conference on Computer Vision · cited 164 times · Not summarized yet

   Abstract

   Recent work has shown that the accuracy of machine learning models can vary substantially when evaluated on a distribution that even slightly differs from that of the training data. As a result, predicting model performance on previously unseen distributions without access to labeled data is an important challenge with implications for increasing the reliability of machine learning models. In the context of distribution shift, distance measures are often used to adapt models and improve their performance on new domains, however accuracy estimation is seldom explored in these investigations. Our investigation determines that common distributional distances such as Frechet distance or Maximum …

2. Close A method it uses Not relevant

   Leveraging Unlabeled Data to Predict Out-of-Distribution Performance

   S. Garg, Sivaraman Balakrishnan, Zachary Chase Lipton et al. · 2022 · International Conference on Learning Representations · cited 205 times · Not summarized yet

   Abstract

   Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. ATC outperforms previous methods across several model architectures, types of distribution shifts (e.g., due to synthetic corruptions, dataset reproduction, or novel s…

3. Close Not relevant

   A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

   Anastasios Nikolas Angelopoulos, Stephen Bates · 2021 · arXiv.org · preprint, not peer reviewed · cited 1,443 times · Not summarized yet

   Abstract

   Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 9…

4. Close Not relevant

   Conformal Prediction Under Covariate Shift

   R. Tibshirani, R. F. Barber, E. Candès et al. · 2019 · Neural Information Processing Systems · cited 969 times · Not summarized yet

   Abstract

   We extend conformal prediction methodology beyond the case of exchangeable data. In particular, we show that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between these two distributions is known---or, in practice, can be estimated accurately with access to a large set of unlabeled data (test covariate points). Our weighted extension of conformal prediction also applies more generally, to settings in which the data satisfies a certain weighted notion of exchangeability. We discuss other potential applications of our new conformal meth…

5. Related Not relevant

   Distribution-free uncertainty quantification for classification under label shift

   A. Podkopaev, Aaditya Ramdas · 2021 · Conference on Uncertainty in Artificial Intelligence · cited 153 times · Not summarized yet

   Abstract

   Trustworthy deployment of ML models requires a proper measure of uncertainty, especially in safety-critical applications. We focus on uncertainty quantification (UQ) for classification problems via two avenues -- prediction sets using conformal prediction and calibration of probabilistic predictors by post-hoc binning -- since these possess distribution-free guarantees for i.i.d. data. Two common ways of generalizing beyond the i.i.d. setting include handling covariate and label shift. Within the context of distribution-free UQ, the former has already received attention, but not the latter. It is known that label shift hurts prediction, and we first argue that it also hurts UQ, by showing de…

Papers that cite it

1. Close Not relevant

   Assessing generalizability of an AI-based visual test for cervical cancer screening

   Syed Rakin Ahmed, Didem Egemen, Brian Befano et al. · 2023 · medRxiv · cited 18 times · Not summarized yet

   Abstract

   A number of challenges hinder artificial intelligence (AI) models from effective clinical translation. Foremost among these challenges are: (1) reproducibility or repeatability, which is defined as the ability of a model to make consistent predictions on repeat images from the same patient taken under identical conditions; (2) the presence of clinical uncertainty or the equivocal nature of certain pathologies, which needs to be acknowledged in order to effectively, accurately and meaningfully separate true normal from true disease cases; and (3) lack of portability or generalizability, which leads AI model performance to differ across axes of data heterogeneity. We recently investigated the …

2. Related Not relevant

   Does confidence calibration improve conformal prediction?

   Hua-Jun Xi, Jianguo Huang, Kangdao Liu et al. · 2024 · Trans. Mach. Learn. Res. · cited 17 times · Not summarized yet

   Abstract

   Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Previous works often employ temperature scaling to calibrate classifiers, assuming that confidence calibration benefits conformal prediction. However, the specific impact of confidence calibration on conformal prediction remains underexplored. In this work, we make two key discoveries about the impact of confidence calibration methods on adaptive conformal prediction. Firstly, we empirically show that current confidence calibration methods (e.g., temperature scaling) typically lead to larger prediction sets in adaptiv…

3. Related Not relevant

   On Temperature Scaling and Conformal Prediction of Deep Classifiers

   Lahav Dabah, Tom Tirer · 2024 · International Conference on Machine Learning · cited 20 times · Not summarized yet

   Abstract

   In many classification applications, the prediction of a deep neural network (DNN) based classifier needs to be accompanied by some confidence indication. Two popular approaches for that aim are: 1) Calibration: modifies the classifier's softmax values such that the maximal value better estimates the correctness probability; and 2) Conformal Prediction (CP): produces a prediction set of candidate labels that contains the true label with a user-specified probability, guaranteeing marginal coverage but not, e.g., per class coverage. In practice, both types of indications are desirable, yet, so far the interplay between them has not been investigated. Focusing on the ubiquitous Temperature Scal…

4. Related Not relevant

   Deploying clinical machine learning? Consider the following

   Charles Lu, Kenglun Chang, Praveer Singh et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

   Abstract

   Despite the intense attention and considerable investment into clinical machine learning research, relatively few applications have been deployed at a large-scale in a real-world clinical environment. While research is important in advancing the state-of-the-art, translation is equally important in bringing these techniques and technologies into a position to ultimately impact healthcare. We believe a lack of appreciation for several considerations are a major cause for this discrepancy between expectation and reality. To better characterize a holistic perspective among researchers and practitioners, we survey several practitioners with commercial experience in developing CML for clinical de…

5. Related* Not relevant

   Multi-scale feature fusion and reweighting based domain generalization for mammogram classification

   Yang Gao, Jiqun Chen, Wenzong Jiang et al. · 2026 · Multimedia Systems · Not summarized yet