A/B Test Sample Size & Power
Calculate required sample size for A/B tests with statistical power and MDE analysis. Enter values for instant results with step-by-step formulas.
Formula
n = [Z_(ฮฑ/2)ยทฯ_pooled + Z_ฮฒยทฯ_control]ยฒ / ฮดยฒ
We use the standard frequentist sample size formula where Z_(ฮฑ/2) represents the critical value for significance (Type I error), Z_ฮฒ represents power (Type II error), ฯ represents standard deviation, and ฮด is the minimum detectable difference. The result is the number of visitors needed per variation.
Worked Examples
Example 1: E-commerce Checkout
Problem:Baseline 2%, MDE 10% (Relative), Power 80%, Sig 5%
Solution:2% -> 2.2% (0.2 percentage-point lift). Using the calculator's two-variant frequentist sample-size formula, you need about 80,706 visitors per variation to reliably detect that change.
Result:161,412 Total Visitors
Example 2: High Traffic Landing Page
Problem:Baseline 15%, MDE 5% (Relative), Power 90%, Sig 5%
Solution:15% -> 15.75% (0.75 percentage-point lift). At 90% power and 95% confidence, the calculator estimates roughly 48,622 visitors per variation.
Result:97,244 Total Visitors
Frequently Asked Questions
What is Minimum Detectable Effect (MDE)?
MDE is the smallest improvement you care about detecting. If you set MDE to 10%, you are calculating sample size to detect at least a 10% lift. Smaller MDEs require much larger sample sizes.
Why does 80% Power matter?
Power (1 - ฮฒ) is the probability of correctly finding a winner if one actually exists. 80% Power means you have a 20% chance of missing a real winner (Type II error). 80% is the industry standard balance between risk and traffic needs.
Can I stop the test early?
No! Stopping as soon as you see significance ('peeking') dramatically increases your False Positive rate. You must commit to the sample size beforehand.
What if I have uneven traffic split?
This assumes a 50/50 split. If you split 90/10, your total sample size requirement increases significantly because the smaller group limits statistical power.
How long should I run the test?
Divide the Total Sample Size by your daily traffic. Ideally, run for full business cycles (e.g., 2 full weeks) to account for day-of-week variances.
Is Bayesian better?
Bayesian methods don't use fixed sample sizes in the same way and allow for optional stopping, but Frequentist (A/B Test Sample Size & Power) is standard for pre-test planning.
Background & Theory
Understanding Sample Size Calculations
Before running an experiment, you must know how much data you need. This prevents two failures: "Underpowered" tests that miss real wins, and "Overpowered" tests that waste traffic on obvious losers.
Key Variables
- Baseline Conversion Rate (p): Your current performance. Tests on low-conversion pages (e.g., 1% checkout) require huge samples compared to high-conversion pages (e.g., 20% click-through).
- Minimum Detectable Effect (MDE): The sensitivity of your "statistical metal detector." If you want to find tiny flakes of gold (1% lift), you need a powerful detector (huge sample). If you only care about giant nuggets (50% lift), a weak detector (small sample) works.
- Statistical Power (1-ฮฒ): Your insurance policy against missing a win. 80% is standard. 95% is for critical tests where missing a win is disastrous.
- Significance Level (ฮฑ): Your protection against lying to yourself. 5% means you accept a 1 in 20 chance of a False Positive.
Practical Tips
- Be Realistic with MDE: Don't set MDE to 1% if you don't have millions of visitors. You'll never finish the test.
- Check Seasonality: Ensure your calculated duration covers full weeks (e.g., 14 days, not 10) to avoid weekend bias.
- Traffic vs Sample: Sample Size is users *entering the experiment*, not total site traffic.
- One Test at a Time: Overlapping tests on the same audience can pollute data unless you use advanced isolation frameworks.
When NOT to use this
- For continuous metrics (Revenue, Time on Site) - use a t-test calculator.
- For multi-armed bandits - they manage traffic dynamically.
- For extremely low traffic sites - A/B testing may not be statistically viable at all.
History
Origins of A/B Testing
A/B testing, or split testing, traces its roots to the 1920s with Sir Ronald Fisher's work on randomized controlled trials in agriculture. In the digital realm, it emerged in the 1990s but became mainstream with Google famously testing 41 shades of blue around 2009.
The Power Analysis Revolution
Initially, digital marketers just "ran tests until they looked green." This led to massive false positives. The application of rigorous statistical power analysisโborrowed from clinical trialsโbecame best practice in the 2010s. Tools like Optimizely and VWO integrated these calculators to stop users from drawing bad conclusions.
Modern Usage
Today, "pre-registration" of sample size is a key tenet of scientific CRO (Conversion Rate Optimization). Companies like Netflix, Amazon, and Booking.com run thousands of tests, all governed by these statistical constraints to ensure business decisions are data-driven, not noise-driven.
Common Misconceptions
- Myth: "I can stop if it's significant after 1 day." Reality: This is the "Peeking Problem." You inflate your error rate.
- Myth: "Any lift is good." Reality: If your MDE is too small, the ROI of the change might not justify the testing cost.