# Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift

**Authors:** Stephan Rabanser, Stephan Günnemann, Zachary C. Lipton  
**arXiv:** [1810.11953](https://arxiv.org/abs/1810.11953)  
**Date:** 2018-10-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1810.11953v1/

## Key points

- The paper explores the problem of building robust and reliable machine learning systems that can detect and handle dataset shifts effectively.
- ML systems tend to fail silently due to their strong dependence on properties of their inputs, such as the i.i.d. assumption.
- The study investigates methods for detecting dataset shift and identifying exemplars that most typify the shift.
- Various perturbations are introduced to both covariates and label distributions in several datasets.
- Classifier-based methods perform well in high-data settings but poorly in low-data settings.
- A two-sample-testing-based approach using pretrained classifiers for dimensionality reduction consistently performs best across the explored dataset shifts.
- Developing ML systems that can detect and handle dataset shifts effectively is important for improving robustness and reliability when faced with unexpected inputs.

## Summary

The paper titled "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift" explores the problem of building robust and reliable machine learning (ML) systems that can detect and handle dataset shifts effectively. Unlike well-designed software systems that fire off warnings when faced with unexpected inputs, ML systems tend to fail silently due to their strong dependence on properties of their inputs, such as the i.i.d. assumption. The study investigates methods for detecting dataset shift and identifying exemplars that most typify the shift. The researchers focus on several datasets and introduce various perturbations to both covariates and label distributions, considering varying magnitudes and fractions of data affected. Interestingly, the study reveals that classifier-based methods perform well in high-data settings but poorly in low-data settings. However, a two-sample-testing-based approach using pretrained classifiers for dimensionality reduction consistently performs best across the explored dataset shifts. Overall, this research sheds light on the importance of developing ML systems that can detect and handle dataset shifts effectively by understanding how different methods perform under various conditions. By doing so, researchers and practitioners can improve the robustness and reliability of ML systems when faced with unexpected inputs.