# All about sample-size calculations for A/B testing: Novel extensions and practical guide

**Authors:** Jing Zhou, Jiannan Lu, Anas Shallah  
**arXiv:** [2305.16459](https://arxiv.org/abs/2305.16459)  
**Date:** 2023-05-25  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2305.16459v2/

## Key points

- Accurate sample size estimation is crucial for trustworthy and efficient online A/B testing
- Existing literature lacks focus on sample size calculation, leading to potential repercussions such as ineffective analysis methods and costly decisions
- Recent paper by Jing Zhou, Jiannan Lu, and Anas Shallah introduces novel extensions and practical guide on sample-size calculations for A/B testing
- Develops new methods for calculating sample sizes in scenarios involving correlated data and addresses nuances of absolute versus relative treatment effects
- Discusses determining minimal observed difference required for statistical significance and its relationship with average treatment effect and sample size calculations
- Provides mathematical proofs, illustrative examples, and simulations to showcase efficacy of proposed methods
- Offers insights into best practices concerning various practical aspects of sample size calculation and experimental design
- Fills a fundamental void in current literature and equips researchers with valuable tools for conducting more robust online A/B testing experiments

## Summary

In the realm of online A/B testing, accurate sample size estimation is crucial for ensuring trustworthy and efficient experimentation. While existing literature covers challenges and best practices in this area, there is a noticeable gap in studies focusing on sample size calculation. This deficiency can have significant repercussions; for instance, underestimating sample size can render meticulous analysis methods ineffective in detecting true significant improvements. This could lead to misguided and costly decisions. To address this critical gap head-on, a recent paper by authors Jing Zhou, Jiannan Lu, and Anas Shallah introduces novel extensions and a practical guide on sample-size calculations for A/B testing. The research not only develops new methods for calculating sample sizes in scenarios involving correlated data but also tackles the nuances of absolute versus relative treatment effects – common occurrences in online experiments. Furthermore, the paper delves into determining the minimal observed difference required for statistical significance and its relationship with average treatment effect and sample size calculations. To support their findings, the authors provide mathematical proofs, illustrative examples, and simulations that showcase the efficacy of their proposed methods. Additionally, they offer insights into best practices concerning various practical aspects of sample size calculation and experimental design. This comprehensive approach not only fills a fundamental void in current literature but also equips researchers and practitioners with valuable tools to conduct more robust and reliable online A/B testing experiments. Accepted by CIKM'23, this research promises to enhance the quality and validity of future online experimentation endeavors.