A/B Test Duration Estimator

Split test duration estimation applies statistical power analysis to determine how long an experiment must run to reliably detect a given effect size, enabling proper experiment planning. ## Concept Overview Duration estimation prevents two common mistakes: stopping tests too early (false positives from noise) and running tests too long (wasted time when significance is impossible). Proper planning sets expectations and enables resource allocation. The key insight is that sample size determines what effects you can detect. Small samples can detect large effects. Detecting small effects requires large samples. Your traffic and desired MDE determine duration. Power analysis formalizes this relationship. Given baseline conversion rate, minimum detectable effect, desired power, and significance level, the required sample size can be calculated before the test begins. ## Key Variables & Intuition • **Daily visitors** — Traffic available for the test • **Baseline conversion** — Current conversion rate being tested • **MDE** — Smallest effect worth detecting • **Power** — Probability of detecting real effects (80-90%) • **Significance** — Confidence level for results (95%) • **Variants** — Number of test variations including control ## Assumptions • Traffic is constant (or average is representative) • Conversion rate is stable • Visitors are independent • No seasonal effects during test period • Proper randomization ## Limitations & Edge Cases • **Seasonality** — Tests spanning holidays may be confounded • **Traffic fluctuations** — Variable traffic changes duration • **Novelty effects** — Initial response may not persist • **Selection bias** — Test audience may not represent population • **Multiple metrics** — Primary metric duration may differ from secondary **Scenario:** A test is planned for 14 days based on conversion rate. But conversion happens 7 days after first visit (delayed attribution). The 14-day test only captures conversions from the first week's visitors. Actual duration should be 21 days (14 + 7-day attribution window). ## Interpretation Guide **Test Duration:** - Under 7 days: Too short; day-of-week effects not captured - 7-14 days: Quick test; ensure sufficient traffic - 14-30 days: Standard duration; captures weekly patterns - 30-60 days: Long test; consider business constraints - Over 60 days: Very long; evaluate if test is practical **Sample Size Drivers:** - Higher baseline = fewer samples needed - Smaller MDE = more samples needed - Higher power = more samples needed - More variants = more samples needed ## Practical Tips • **Plan before starting** — Calculate duration, don't just run and hope • **Run full weeks** — Day-of-week patterns are real; complete weeks prevent bias • **Avoid peeking** — Multiple peeks inflate false positive rate • **Consider practical significance** — Statistically significant ≠ worth implementing • **Document decisions** — Record MDE, power, expected duration before launch • **Use proper tools** — Platform calculators or statistical packages • **Account for exclusions** — Bots, returning users affect effective sample • **Plan for delays** — Traffic may be lower than expected • **Commit to the plan** — Don't extend or shorten based on interim results • **Consider opportunity cost** — Very long tests may not be worth it ## Common Mistakes • **Stopping early** — Declaring winners before reaching sample size • **Ignoring power** — Underpowered tests miss real effects • **Too small MDE** — Wanting to detect tiny effects leads to impractical durations • **Forgetting variants** — Each variant needs full sample size • **Ignoring seasonality** — Testing through Black Friday skews results • **No baseline measurement** — Inaccurate baseline leads to wrong duration estimate • **Peeking at results** — Inflates false positive rate • **Stopping for business reasons** — If you stop early, treat results as directional only ## When NOT to Use • **Qualitative decisions** — Some changes don't need statistical validation • **Obvious improvements** — Fixing a broken button doesn't need a test • **Very low traffic** — Tests may be impractical; use other methods • **Rapid iteration** — Sometimes shipping fast and monitoring is better

A/B test duration estimation evolved from basic significance testing through sample size planning to sophisticated experiment design frameworks that balance statistical rigor with business velocity. ## Origins & Why It Emerged Statistical hypothesis testing dates to the early 20th century (Fisher, Neyman, Pearson). But applying statistics to online experiments became practical only with web analytics in the 2000s. Early web testing was often underpowered—teams ran tests too short, declared winners based on noise, and made changes that didn't actually improve metrics. The replication crisis in academia highlighted similar problems. Sample size planning emerged as a discipline. Rather than stopping when significance appeared, teams learned to calculate required samples beforehand and commit to running until that threshold. ## Evolution in Practice Simple calculators enabled basic sample size calculation. Given baseline rate, desired MDE, power, and significance, anyone could determine required samples. This was a major improvement over ad hoc testing. Sequential testing methods (Bayesian approaches, group sequential designs) offered alternatives to fixed-sample tests. These allow valid early stopping but require different statistical frameworks and careful implementation. Platform sophistication increased. Optimizely, VWO, and Google Optimize built statistics into their products. Automated duration recommendations made proper testing accessible to non-statisticians. ## Modern Usage Today Modern experimentation platforms automate duration calculation and prevent premature stopping. They integrate with traffic forecasting to estimate test completion dates and alert when tests are underpowered. Multi-armed bandit approaches offer alternatives to classical A/B testing, optimizing during the experiment rather than just evaluating at the end. These require different duration considerations. Meta-analysis of multiple experiments, pre-registration of experiment plans, and holdout groups for validation have become standard at sophisticated organizations. ## Common Misconceptions • **Bigger sample is always better** — Diminishing returns after sufficient power; opportunity cost of long tests • **Significance means the effect is real** — It means unlikely due to chance; could still be noise • **I can stop when significant** — Only if using sequential methods designed for early stopping • **More variants = more learning** — Also means longer tests and multiple comparison problems • **One week is enough** — Depends on traffic, conversion rate, and MDE; rarely is one week sufficient for low-traffic sites