Experiment Power & Sample Size Checker
Calculate A/B test sample size, statistical power, and test duration. Validate experiment feasibility with MDE analysis and power curves.
Formula
n = [z_α√(2p̄(1-p̄)) + z_β√(p₁(1-p₁) + p₂(1-p₂))]² / (p₂-p₁)²
The sample size formula balances the competing demands of significance and power. The z-scores (z_α and z_β) represent the critical values for the desired significance level and power. The pooled proportion (p̄) estimates variance under the null hypothesis, while the individual proportions (p₁, p₂) estimate variance under the alternative. The denominator is the squared effect size—smaller effects require exponentially larger samples. This formula works because it quantifies the signal-to-noise ratio needed to distinguish true effects from random variation. The numerator captures noise (variance), and the denominator captures signal (effect size). Adequate sample size ensures the signal exceeds noise with the specified confidence.
Worked Examples
Example 1: E-commerce Checkout Optimization
Problem:An e-commerce site has 5% checkout conversion, 50,000 daily visitors, and wants to detect a 10% relative improvement. Calculate sample size and test duration at 95% significance and 80% power.
Solution:Parameters: - Baseline: 5% conversion - MDE: 10% relative (5% → 5.5% absolute) - α = 0.05, Power = 80% - Daily traffic: 50,000 Calculation: - Absolute effect: 5% × 10% = 0.5 percentage points - z_α/2 = 1.96 (two-tailed, 95%) - z_β = 0.84 (80% power) - Pooled p = (0.05 + 0.055) / 2 = 0.0525 Sample size per variant: n = [z_α√(2p̄(1-p̄)) + z_β√(p1(1-p1) + p2(1-p2))]² / (p2-p1)² n ≈ 31,000 per variant Total sample: 62,000 Days needed: 62,000 / 50,000 = 1.24 days ⚠️ This seems too short. Let's verify: - With 50K daily traffic to checkout page: unlikely - More realistic: 50K site visitors, ~5% reach checkout = 2,500/day - Revised days: 62,000 / 2,500 = 25 days Feasibility: Good (25 days) Sanity Check: - 10% MDE on 5% baseline is reasonable - 25 days covers multiple weeks (accounts for week
Result:31K per variant | 62K total | 25 days (checkout traffic) | Good feasibility
Example 2: Low-Traffic SaaS Pricing Test
Problem:A B2B SaaS has 500 daily trial signups, 2% trial-to-paid conversion, and wants to test a pricing change. What MDE is realistic? They want the test done in 4 weeks.
Solution:Constraint-Based Planning: - Available sample in 28 days: 500 × 28 = 14,000 - Per variant (2 variants): 7,000 - Baseline: 2% conversion - Target duration: 4 weeks Reverse-Calculate MDE: Given n = 7,000 per variant, what MDE is detectable? Using power formula rearranged: MDE = f(n, baseline, α, power) With 2% baseline and 7,000 samples: - At 80% power, 95% significance - Detectable absolute effect ≈ 0.7 percentage points - Relative MDE ≈ 35% (2% → 2.7%) Interpretation: You can only reliably detect a 35%+ relative improvement. Is this acceptable? - If pricing change is expected to have 30%+ impact: Yes - If you need to detect 10% improvement: No (need 63,000 samples = 18 weeks) Recommendations: 1. Accept 35% MDE if pricing change is substantial 2. Extend test to 8 weeks for ~20% MDE 3.
Result:7K per variant in 4 weeks | Only detects 35%+ MDE | Consider 8 weeks for 20% MDE
Example 3: Mobile App Feature Launch
Problem:A mobile app tests a new feature. DAU: 200,000. Baseline engagement: 15% use feature X. Want to detect 5% relative lift at 95%/80%. Traffic split: 50/50.
Solution:High-Traffic Scenario: - Baseline: 15% feature usage - MDE: 5% relative (15% → 15.75%) - Absolute effect: 0.75 percentage points - DAU: 200,000 - Split: 50/50 (100K per variant daily) Sample Size Calculation: With 15% baseline (high rate), variance is higher: - p(1-p) = 0.15 × 0.85 = 0.1275 n = [1.96√(2×0.15375×0.84625) + 0.84√(0.15×0.85 + 0.1575×0.8425)]² / (0.0075)² n ≈ 55,000 per variant Total: 110,000 Days needed: 110,000 / 200,000 = 0.55 days ⚠️ Sub-day result suggests we should: 1. Run for minimum 1-2 weeks anyway (weekly patterns, novelty) 2. Consider smaller MDE since we have traffic headroom Optimized Plan: - Run for 14 days minimum (industry best practice) - Available sample: 200K × 14 = 2.8M - Per variant: 1.4M - Detectable MDE: ~1% relative (extremely precise) Recommenda
Result:55K per variant needed | <1 day for sample | Run 2 weeks minimum for validity
Frequently Asked Questions
What is statistical power in A/B testing?
Statistical power (1-β) is the probability of detecting a real effect when it exists. 80% power means if a true effect exists, you have 80% chance of detecting it. Low power means high false negative risk—you might miss real improvements.
What is Minimum Detectable Effect (MDE)?
MDE is the smallest effect size your experiment can reliably detect. A 10% MDE means you can detect if the variant is 10%+ better than control. Smaller MDEs require larger samples. Choose MDE based on what effect size would be business-meaningful.
How do I calculate sample size for A/B tests?
Sample size depends on: baseline conversion rate, MDE, significance level (α), and power (1-β). The formula involves z-scores and pooled variance. Higher baseline rates and larger MDEs require smaller samples; lower α and higher power require larger samples.
What significance level should I use?
95% (α=0.05) is standard, meaning 5% false positive risk. Use 99% for high-stakes tests (pricing, major UX changes). Some teams use 90% for exploratory tests. Never go below 90%—you're just fooling yourself with noise.
Why does my test need so many samples?
Small effects and low baseline rates require large samples. If you're trying to detect a 2% improvement on a 1% conversion rate, you need hundreds of thousands of visitors. Consider: is detecting such small effects worth the time? Focus on bigger swings.
Can I stop a test early if results look significant?
Peeking and stopping early inflates false positives dramatically. If you peek 10 times at α=0.05, your true false positive rate might be 25%+. Use sequential testing methods (like group sequential or always-valid p-values) if you need early stopping.
What's the difference between one-tailed and two-tailed tests?
Two-tailed tests detect effects in either direction (better or worse). One-tailed tests only detect improvement. Use two-tailed unless you truly don't care about detecting harm. One-tailed requires ~20% fewer samples but is often misused.
How do I handle multiple variants?
Multiple variants increase false positive risk. Apply corrections: Bonferroni divides α by number of comparisons (conservative). Benjamini-Hochberg controls false discovery rate (less conservative). Or use holdout groups and sequential testing.
What if my traffic is too low for the MDE I want?
Options: accept larger MDE, test on higher-traffic pages, run longer (but beware novelty effects), use variance reduction techniques (CUPED), or combine multiple metrics. Sometimes the business answer is 'this test isn't feasible.'
How do novelty and learning effects affect tests?
Novelty effects inflate early results—users engage more with anything new. Learning effects mean users need time to adapt. Run tests for full business cycles (min 1-2 weeks) and exclude the first few days from analysis if novelty is suspected.
Background & Theory
Experiment power and sample size analysis ensures A/B tests have sufficient statistical precision to detect meaningful effects while avoiding wasted resources on impractical test designs.
Concept Overview
Statistical power is the probability of detecting a true effect when it exists (1 - Type II error rate). An underpowered experiment risks missing real improvements—the variant might actually be better, but the test fails to detect it due to noise.
Sample size calculation balances precision against practicality. Smaller minimum detectable effects (MDE) require larger samples, meaning longer test durations. The right MDE is one that represents a business-meaningful improvement and is feasible given available traffic.
Power analysis should happen before launching experiments, not after. Pre-registration of sample size, duration, and success criteria prevents p-hacking and post-hoc rationalization. It also prevents wasting resources on tests that can never reach statistical validity.
Key Variables & Intuition
- Baseline conversion rate — Higher baselines mean higher variance; moderate rates (10-30%) are easiest to test
- Minimum Detectable Effect (MDE) — Smallest effect worth detecting; smaller MDE = larger samples
- Significance level (α) — False positive risk, typically 5%; lower α requires more samples
- Statistical power (1-β) — True positive probability, typically 80%; higher power requires more samples
- Daily traffic — Determines test duration; more traffic = faster tests
- Test type — One-tailed vs two-tailed; two-tailed is standard and conservative
Assumptions
- Traffic is stable over the test period
- Visitors are randomly assigned to variants
- No interaction between variants (spillover)
- Metric follows approximately normal distribution (large samples)
- Effect is constant over time (no novelty/learning effects)
Limitations & Edge Cases
- Ratio metrics — Conversion rates are ratios; variance estimation is more complex
- Revenue metrics — Heavy-tailed distributions require special handling
- Network effects — Users influencing each other violates independence
- Low traffic — Small samples require exact tests, not normal approximations
- Novelty effects — Early results may not represent long-term effect
Scenario: A test shows 15% lift after 3 days with p=0.02. Should you ship? Not necessarily. Short tests are susceptible to novelty effects and don't capture weekly patterns. The early significance might be noise that would regress. Wait for the pre-planned duration regardless of early results.
Interpretation Guide
Feasibility Assessment:
- < 14 days: Excellent — proceed confidently
- 14-30 days: Good — plan for the duration
- 30-60 days: Moderate — consider larger MDE or higher traffic
- 60-90 days: Challenging — question whether the test is worthwhile
- > 90 days: Impractical — rethink the test or find alternatives
MDE Guidance:
- 1-5%: Very precise, requires massive traffic
- 5-10%: Standard for high-traffic sites
- 10-20%: Reasonable for medium traffic
- 20%+: Only detects large effects
Practical Tips
- Calculate before testing — Never start a test without knowing required sample size
- Plan for full duration — Don't peek and stop early; commit to the plan
- Account for patterns — Include full business cycles (weeks, months if relevant)
- Use variance reduction — CUPED can cut required samples by 30-40%
- Consider opportunity cost — 90-day tests block testing other ideas
- Document assumptions — Record baseline, MDE, and rationale
- Validate sample size tools — Different tools give different numbers; understand why
- Plan for multiple testing — If testing many variants, adjust significance levels
Common Mistakes
- Stopping early on significance — Inflates false positive rate dramatically
- Ignoring power — Underpowered tests waste resources and miss effects
- Unrealistic MDE — Hoping to detect 1% lift with 1000 samples
- Forgetting about baseline — Low conversion rates require much larger samples
- One-tailed without justification — Gaming the statistics
- Not accounting for multiple variants — Each comparison adds false positive risk
- Ignoring practical duration — Sample size without traffic context is meaningless
- Post-hoc power calculations — Meaningless after seeing results
When NOT to Use
- For observational data — A/B test formulas assume randomization
- With very small samples — Exact tests or Bayesian methods are better
- For non-independent observations — Network effects require different methods
- When baseline is unknown — Estimate baseline first with observational data
History
Statistical power analysis evolved from agricultural and medical research foundations through academic statistics to become essential infrastructure for modern digital experimentation.
Origins & Why It Emerged
The foundations trace to Ronald Fisher's work in agricultural experiments (1920s-30s). Fisher developed ANOVA and significance testing to determine if crop treatments had real effects. His significance level (p < 0.05) became the threshold still used today.
Jerzy Neyman and Egon Pearson formalized power analysis in the 1930s. They introduced Type I errors (false positives) and Type II errors (false negatives), and the concept of power as the probability of correctly rejecting a false null hypothesis.
Jacob Cohen's "Statistical Power Analysis for the Behavioral Sciences" (1969, revised 1988) brought power analysis to social sciences. His work established effect size conventions (small, medium, large) and made power calculation accessible to researchers without deep statistical training.
How It Evolved in Practice
Medical trials drove rigorous power analysis. Underpowered drug studies waste resources and potentially harm patients. Regulatory agencies (FDA) began requiring power calculations in trial protocols. The clinical trials community developed sample size formulas for various designs.
The digital experimentation revolution (2000s-present) democratized A/B testing. Companies like Google, Amazon, and Microsoft ran thousands of experiments simultaneously. This scale revealed how often underpowered tests led to wrong decisions.
Experimentation platforms (Optimizely 2010, VWO, Google Optimize) built power calculators into their products. What was once specialist statistical work became accessible to product managers and marketers.
Modern Usage Today
Modern experimentation platforms provide real-time power estimates. As tests run, they show projected completion dates and current power. This prevents premature stopping and helps prioritize test queues.
Variance reduction techniques (CUPED, regression adjustment) improve power without more samples. By controlling for pre-experiment covariates, experiments need 20-40% fewer samples for the same power.
Multi-armed bandits and adaptive designs optimize the exploration-exploitation tradeoff. Rather than fixed sample sizes, these methods allocate traffic dynamically while maintaining statistical validity.
Common Misconceptions Historically
- "Significant result means the effect is large" — Significance and effect size are different; with huge samples, tiny effects become significant
- "80% power is always sufficient" — 80% is convention, not law; high-stakes decisions may warrant 90%+ power
- "Power doesn't matter once you see significance" — Low power means significant results are more likely to be inflated (winner's curse)
- "More samples are always better" — Diminishing returns exist; also longer tests risk novelty effects and context changes
- "Sample size calculators give exact answers" — They're estimates based on assumptions; actual required samples vary