# An Automatic Finite-Sample Robustness Metric: Can Dropping a Little Data Change Conclusions?

**Authors:** Tamara Broderick, Ryan Giordano, Rachael Meager  
**arXiv:** [2011.14999](https://arxiv.org/abs/2011.14999)  
**Date:** 2020-11-30  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2011.14999v1/

## Key points

- Authors: Tamara Broderick, Ryan Giordano, Rachael Meager
- Proposed method: Approximate Maximum Influence Perturbation
- Purpose: Evaluate sensitivity of econometric analyses to exclusion of small sample portion
- Applicability: OLS, IV, GMM, MLE, variational Bayes estimators
- Automatically computable
- Provides finite-sample error bounds for linear and instrumental variables regressions
- Identifies influential observations that can impact study conclusions if omitted

## Summary

In their paper titled "An Automatic Finite-Sample Robustness Metric: Can Dropping a Little Data Change Conclusions? ", authors Tamara Broderick, Ryan Giordano, and Rachael Meager propose a novel method to evaluate the sensitivity of econometric analyses to the exclusion of a small portion of the sample. The authors introduce a finite-sample metric known as the Approximate Maximum Influence Perturbation to approximate the number or fraction of observations that exert the most significant influence on a given result when removed from the analysis. This metric is designed to be automatically computable and applicable to various common estimators such as OLS, IV, GMM, MLE, and variational Bayes. The authors provide explicit finite-sample error bounds for linear and instrumental variables regressions, offering a precise lower bound on sensitivity for any estimator at minimal computational cost. Through empirical applications, Broderick et al. demonstrate that even simple 2-parameter linear regression analyses of randomized trials can exhibit high sensitivity to data perturbations. They also show that dropping less than 1% of the sample can lead to a change in the sign of a treatment effect in certain cases, despite small standard errors. This highlights the importance of assessing robustness in econometric analyses and provides a valuable tool in identifying influential observations that may significantly impact study conclusions when omitted from the analysis.