Search Results for « Dataset Shift »

Dataset shift occurs when there is a discrepancy between the distribution of samples used for training and testing a model, which can lead to a reduction in the model’s predictive performance when attempting to generalize model predictions. Dataset shift can occur with respect to the 1) criterion variable (prior probability shift), 2) predictor variables (covariate shift), and 3) the relationship between criterion and predictors (concept shift). The present paper investigated the implications of avoiding dataset shift during k-fold cross-validation. To circumvent the various forms of dataset shift during cross-validation, a range of anticlustering algorithms were employed to ensure equal means, variances, and covariance structure across folds. A comparative analysis was conducted to assess the bias in validation error obtained through anticlustering-based cross-validation and standard cross-validation.The bias in validation error was computed using the true test error based on unseen test data as reference. Utilizing linear regression models, simulated and empirical datasets demonstrated that avoiding prior probability shift in conjunction with covariate shift—but not concept shift—can effectively curtail the bias in the prediction error during cross-validation. The advantage of using anticlustering-based cross-validation over standard cross-validation however diminished when R2 increased. The study underscores the merits of leveraging anticlustering methodologies within the framework of k-fold cross-validation to mitigate adverse effects of prior probability and covariate shift by equating means and variances in predictor and criterion variables across folds.

Objective: Artificial intelligence (AI) systems for cancer diagnosis often demonstrate strong performance in controlled settings but fail to generalize reliably in real-world clinical environments due to data heterogeneity, dataset shift, and lack of uncertainty awareness. This study proposes a framework designed to address these limitations by aligning diagnostic predictions with clinically meaningful decision-making.Methods: We introduce CaliFusion-CNN, a multimodal deep learning framework that integrates imaging data with structured clinical variables through modality-specific encoders and a fusion mechanism. The framework incorporates post-hoc confidence calibration to provide reliable probability estimates and supports threshold-based decision logic. Evaluation was conducted across heterogeneous datasets and experimental conditions using clinically relevant metrics, including sensitivity, specificity, AUROC, predictive values, and agreement measures. Results: The multimodal configuration demonstrated improved diagnostic performance compared to single-modality approaches, achieving higher sensitivity, specificity, and agreement with reference annotations. Confidence calibration enhanced predictive reliability by reducing overconfident errors and improving performance at clinically relevant operating points. Sensitivity analysis showed that multimodal integration and calibration contributed to robustness under variable data conditions, including reduced image quality and incomplete clinical information. Conclusion: CaliFusion-CNN provides a clinically aligned framework for cancer diagnosis that integrates multimodal learning with uncertainty-aware inference and deployment-oriented evaluation. By emphasizing reliability, interpretability, and robustness to dataset shift, the proposed approach supports more trustworthy and actionable AI-assisted decision-making in oncology.

Abstract Purpose To evaluate the external transportability of source-derived probability calibration for chest radiograph deep learning classifiers under distribution shift and determine its dependence on sample size and disease prevalence. Materials and Methods This retrospective secondary computational analysis of public, de-identified radiographic datasets involved no participant contact or new data collection. Cohort sizes were prespecified by availability without prospective power calculation: 106,686 NIH ChestX-ray14 images (29,323 patients; 1992-2015) for development and internal testing, 20,000 CheXpert images (20,000 patients; October 2002-July 2017), and 14,998 VinBigData images (2018-2020) for external testing. ResNet-18, DenseNet-121, and DeiT-Tiny classifiers for cardiomegaly and pleural effusion were trained across four patient fractions and three seeds (36 fits). Calibrators were fitted exclusively on NIH data. Twelve primary external Brier-score contrasts (temperature-scaled versus raw probabilities) were evaluated using paired cluster-bootstrapping with Holm adjustment. Results Temperature scaling reduced Brier scores in nine of 12 primary contrasts but worsened pleural effusion calibration on VinBigData across all architectures (Brier increases: ResNet-18, 0.001460 [95% CI: 0.001347, 0.001568]; DenseNet-121, 0.001193 [95% CI: 0.001109, 0.001274]; DeiT-Tiny, 0.000498 [95% CI: 0.000417, 0.000577]; all adjusted P =.006). Standardizing class weights to NIH prevalence reversed all CheXpert improvements. Although full versus 10% training improved external AUROC across all comparisons, DeiT-Tiny CheXpert pleural effusion Brier score worsened by 0.007039 despite a 0.032466 AUROC gain. At 80% selective image retention, calibrated ensembles retained only 12.3%-17.2% of VinBigData positive pleural effusion examinations. Conclusion Source-fitted calibration is not uniformly transportable across health systems, and discrimination gains do not guarantee calibrated probabilities. External evaluation must simultaneously assess probability reliability, prevalence shifts, and retained-case composition.

Deep neural networks have demonstrated high diagnostic accuracy in dermoscopic skin cancer screening under canonical independent and identically distributed (i.i.d.) assumptions.However, when deployed across distinct healthcare centers, predictive reliability often degrades due to cross-dataset distribution shifts induced by disparate imaging hardware, optical illumination, sensor calibration, and demographic variations. In this paper, we formulatea visual representation framework termed the Domain-Generalized Boundary-AwareTransformer (DG-BAT) to learn domain-invariant morphological representations thatremain discriminative, calibrated, and robust under unseen domain shifts.

Growing concern surrounds the threats faced by Vanuatu’s famous diversity of Indigenouslanguages, with particular attention directed towards the pressure that Bislama exerts as thecountry’s lingua franca. Here, I report the results of exploratory analyses of Vanuatu’s mostrecent national census, leveraging individual-level data for nearly 250,000 people to gauge themagnitude of the shift away from Indigenous languages and identify the factors implicated inthis process. The data reveal that 83.9% of Vanuatu’s population can ‘easily’ speak anIndigenous language and that 74.9% learnt an Indigenous first language. Conversely, 9.6% ofpeople cannot speak an Indigenous language (including 9.4% of Indigenous Ni-Vanuatu), anda severely conservatively-biased estimate shows that Bislama is the first language learnt for atleast 14.2% of the population, including 14.3% of Ni-Vanuatu, with the true figure likely closerto 25%. The youngest generations show considerably poorer Indigenous language abilities andsubstantially greater learning of Bislama as a first language, a pattern consistent with amovement away from Indigenous languages over time. Further patterns that emerge indicatethat Indigenous languages fare best in Vanuatu’s rural areas, in the Area Councils with thesmallest populations, and in the communities that are the least diverse in terms of people’sisland backgrounds. Focusing in on these rural areas, where Bislama likely poses the greatestthreat to Indigenous languages, individuals who have a history of migration, who have accessto modern communication technologies, who live in economically better-off households, andwho live in households without kastom land tenure are less likely to speak an Indigenouslanguage easily and are more likely to learn Bislama as a first language. While strictlycorrelational and non-causal, these exploratory analyses reflect both the vitality of Indigenouslanguages in Vanuatu today and the serious challenges they face from a suite of likelyinterconnected factors.

Abstract Reliable AI-based early warning across courses requires risk probabilities that remain interpretable when learner behavior and outcome prevalence shift. Using the Open University Learning Analytics Dataset, this study develops and evaluates a leakage-controlled cross-module framework that combines temporal behavioral profiling, zero-shot transfer, target-adaptive probability calibration, and capacity-constrained support prioritization. Models were assessed across 28 held-out module-cutoff combinations at days 14, 28, 42, and 56. Earlier presentations supplied learner-disjoint calibration samples covering 10%, 20%, or 30% of the historical target pool, while the chronologically latest presentation was reserved for final evaluation. At the 30% calibration level, the mean Brier score decreased from 0.243 to 0.205, negative log-likelihood from 0.708 to 0.599, expected calibration error from 0.158 to 0.076, and adaptive expected calibration error from 0.161 to 0.078. All seven module-level comparisons favored adaptation for each core metric, with bootstrap intervals excluding zero. Ranking performance was preserved because the selected mappings were predominantly monotonic. Importance weighting and source-side ensemble reweighting did not improve significantly beyond standard target calibration. The results show that modest historical target-domain data can substantially restore probability reliability under cross-course dataset shift and support more interpretable, capacity-aware review.

Abstract Recalibrating a deployed classifier with very few target labels can increase pre diction loss under dataset shift. We study the magnitude of this update through curvature-based shrinkage toward a source-calibrated temperature. A quadratic surrogate gives a classical Gaussian shrinkage interpretation and identifies the assumptions needed for a bias–variance analysis; it does not provide an uncon ditional safety guarantee for the implemented estimator. We compare curvature shrinkage, a significance-gated alternative, and anchored calibration baselines using shared label queries. An archived development benchmark contains 32 domains from eight families. The subsequent three-family evaluation covers sam pling differences in remote sensing, population differences in activity recognition, and calendar-based shifts in restaurant inspections. Implementing the prede clared single-class exclusion leaves nine eligible domains at budgets of 8 and 16 labels and eight at 32 labels. Curvature shrinkage improves mean negative log likelihood by 0.055–0.096 nats over retaining the source calibration. Its additional advantage over a constant anchor is small and statistically inconclusive. At eight labels, gating reduces the frequency of negative gains from 41.5% to 12.6%, but has lower mean gain and a worse minimum domain-average gain. Rankings also depend on the magnitude threshold used to define harm. The results support a deployment trade-off rather than uniform dominance, and motivate explicit reporting of both frequency and severity of harmful recalibration.

Background: Artificial intelligence (AI) has shown promise in breast ultrasound image analysis, but most evidence still comes from single-dataset studies. Clinical translation requires evaluation under heterogeneous acquisition and curation conditions. This study presents a patient-leakage-aware, reproducible benchmark for breast ultrasound AI under dataset shift, with emphasis on external generalization, calibration, and confidence-related behavior. Methods: A reproducible benchmark framework was developed using patient-level splitting, internal testing, pairwise cross-dataset evaluation, whole-image and region-of-interest (ROI) input strategies, calibration analysis, targeted ROI-margin sensitivity analysis, representative explainable AI visualization, and an auxiliary lesion-versus-normal confidence-based analysis. Four public breast ultrasound datasets (BUSI, BUS-UCLM, BUS-BRA, and BrEaST) were harmonized for a primary benign-versus-malignant lesion classification task. Normal images were excluded from the primary endpoint and used only in auxiliary analyses when sufficient numbers were available. Results: Cross-dataset testing was weaker on average than internal testing, with mean raw AUROC decreasing from 0.801 to 0.719 and mean balanced accuracy from 0.723 to 0.635. ROI input improved external performance, especially for the vision transformer, increasing mean external AUROC from 0.666 to 0.805 and mean external balanced accuracy from 0.594 to 0.713 relative to whole-image input. Temperature scaling improved calibration-related metrics, reducing mean external expected calibration error from 0.180 to 0.150 and mean external negative log-likelihood from 0.848 to 0.682. Conclusions: This study establishes a reproducible benchmark for evaluating breast ultrasound AI under dataset shift, with explicit attention to patient-level leakage control, external validity, and reliability of predicted probabilities.

Abstract Machine learning brings the hope of finding new biomarkers extracted from cohorts with rich biomedical measurements. A good biomarker is one that gives reliable detection of the corresponding condition. However, biomarkers are often extracted from a cohort that differs from the target population. Such a mismatch, known as a dataset shift, can undermine the application of the biomarker to new individuals. Dataset shifts are frequent in biomedical research, e.g., because of recruitment biases. When a dataset shift occurs, standard machine-learning techniques do not suffice to extract and validate biomarkers. This article provides an overview of when and how dataset shifts break machine-learning–extracted biomarkers, as well as detection and correction strategies.