Experiment Power & Sample Size Checker

Experiment power and sample size analysis ensures A/B tests have sufficient statistical precision to detect meaningful effects while avoiding wasted resources on impractical test designs. ## Concept Overview Statistical power is the probability of detecting a true effect when it exists (1 - Type II error rate). An underpowered experiment risks missing real improvements—the variant might actually be better, but the test fails to detect it due to noise. Sample size calculation balances precision against practicality. Smaller minimum detectable effects (MDE) require larger samples, meaning longer test durations. The right MDE is one that represents a business-meaningful improvement and is feasible given available traffic. Power analysis should happen before launching experiments, not after. Pre-registration of sample size, duration, and success criteria prevents p-hacking and post-hoc rationalization. It also prevents wasting resources on tests that can never reach statistical validity. ## Key Variables & Intuition • **Baseline conversion rate** — Higher baselines mean higher variance; moderate rates (10-30%) are easiest to test • **Minimum Detectable Effect (MDE)** — Smallest effect worth detecting; smaller MDE = larger samples • **Significance level (α)** — False positive risk, typically 5%; lower α requires more samples • **Statistical power (1-β)** — True positive probability, typically 80%; higher power requires more samples • **Daily traffic** — Determines test duration; more traffic = faster tests • **Test type** — One-tailed vs two-tailed; two-tailed is standard and conservative ## Assumptions • Traffic is stable over the test period • Visitors are randomly assigned to variants • No interaction between variants (spillover) • Metric follows approximately normal distribution (large samples) • Effect is constant over time (no novelty/learning effects) ## Limitations & Edge Cases • **Ratio metrics** — Conversion rates are ratios; variance estimation is more complex • **Revenue metrics** — Heavy-tailed distributions require special handling • **Network effects** — Users influencing each other violates independence • **Low traffic** — Small samples require exact tests, not normal approximations • **Novelty effects** — Early results may not represent long-term effect **Scenario:** A test shows 15% lift after 3 days with p=0.02. Should you ship? Not necessarily. Short tests are susceptible to novelty effects and don't capture weekly patterns. The early significance might be noise that would regress. Wait for the pre-planned duration regardless of early results. ## Interpretation Guide **Feasibility Assessment:** - < 14 days: Excellent — proceed confidently - 14-30 days: Good — plan for the duration - 30-60 days: Moderate — consider larger MDE or higher traffic - 60-90 days: Challenging — question whether the test is worthwhile - > 90 days: Impractical — rethink the test or find alternatives **MDE Guidance:** - 1-5%: Very precise, requires massive traffic - 5-10%: Standard for high-traffic sites - 10-20%: Reasonable for medium traffic - 20%+: Only detects large effects ## Practical Tips • **Calculate before testing** — Never start a test without knowing required sample size • **Plan for full duration** — Don't peek and stop early; commit to the plan • **Account for patterns** — Include full business cycles (weeks, months if relevant) • **Use variance reduction** — CUPED can cut required samples by 30-40% • **Consider opportunity cost** — 90-day tests block testing other ideas • **Document assumptions** — Record baseline, MDE, and rationale • **Validate sample size tools** — Different tools give different numbers; understand why • **Plan for multiple testing** — If testing many variants, adjust significance levels ## Common Mistakes • **Stopping early on significance** — Inflates false positive rate dramatically • **Ignoring power** — Underpowered tests waste resources and miss effects • **Unrealistic MDE** — Hoping to detect 1% lift with 1000 samples • **Forgetting about baseline** — Low conversion rates require much larger samples • **One-tailed without justification** — Gaming the statistics • **Not accounting for multiple variants** — Each comparison adds false positive risk • **Ignoring practical duration** — Sample size without traffic context is meaningless • **Post-hoc power calculations** — Meaningless after seeing results ## When NOT to Use • **For observational data** — A/B test formulas assume randomization • **With very small samples** — Exact tests or Bayesian methods are better • **For non-independent observations** — Network effects require different methods • **When baseline is unknown** — Estimate baseline first with observational data

Statistical power analysis evolved from agricultural and medical research foundations through academic statistics to become essential infrastructure for modern digital experimentation. ## Origins & Why It Emerged The foundations trace to Ronald Fisher's work in agricultural experiments (1920s-30s). Fisher developed ANOVA and significance testing to determine if crop treatments had real effects. His significance level (p < 0.05) became the threshold still used today. Jerzy Neyman and Egon Pearson formalized power analysis in the 1930s. They introduced Type I errors (false positives) and Type II errors (false negatives), and the concept of power as the probability of correctly rejecting a false null hypothesis. Jacob Cohen's "Statistical Power Analysis for the Behavioral Sciences" (1969, revised 1988) brought power analysis to social sciences. His work established effect size conventions (small, medium, large) and made power calculation accessible to researchers without deep statistical training. ## How It Evolved in Practice Medical trials drove rigorous power analysis. Underpowered drug studies waste resources and potentially harm patients. Regulatory agencies (FDA) began requiring power calculations in trial protocols. The clinical trials community developed sample size formulas for various designs. The digital experimentation revolution (2000s-present) democratized A/B testing. Companies like Google, Amazon, and Microsoft ran thousands of experiments simultaneously. This scale revealed how often underpowered tests led to wrong decisions. Experimentation platforms (Optimizely 2010, VWO, Google Optimize) built power calculators into their products. What was once specialist statistical work became accessible to product managers and marketers. ## Modern Usage Today Modern experimentation platforms provide real-time power estimates. As tests run, they show projected completion dates and current power. This prevents premature stopping and helps prioritize test queues. Variance reduction techniques (CUPED, regression adjustment) improve power without more samples. By controlling for pre-experiment covariates, experiments need 20-40% fewer samples for the same power. Multi-armed bandits and adaptive designs optimize the exploration-exploitation tradeoff. Rather than fixed sample sizes, these methods allocate traffic dynamically while maintaining statistical validity. ## Common Misconceptions Historically • **"Significant result means the effect is large"** — Significance and effect size are different; with huge samples, tiny effects become significant • **"80% power is always sufficient"** — 80% is convention, not law; high-stakes decisions may warrant 90%+ power • **"Power doesn't matter once you see significance"** — Low power means significant results are more likely to be inflated (winner's curse) • **"More samples are always better"** — Diminishing returns exist; also longer tests risk novelty effects and context changes • **"Sample size calculators give exact answers"** — They're estimates based on assumptions; actual required samples vary