Abtest Significance Calculator
Test whether your A/B experiment results are statistically significant. Enter control and variation conversion rates and sample sizes to get p-value and
Reviewed by Daniel Agrici, Founder & Lead Developer · Editorial policy
Abtest Significance Calculator Formula
z = (p1 - p2) / sqrt(p_pooled × (1-p_pooled) × (1/n1 + 1/n2))
Uses a two-proportion z-test. The pooled proportion combines both groups. The z-score measures how many standard errors the observed difference is from zero. The p-value is derived from the standard normal distribution (two-tailed). Statistical significance is declared at p < 0.05.
Abtest Significance Calculator — Worked Examples
Example 1: Landing Page A/B Test
Problem:Control: 10,000 visitors, 300 conversions (3.0%). Variant: 10,000 visitors, 360 conversions (3.6%). Is the variant significantly better?
Solution:Lift: (3.6-3.0)/3.0 = 20.0%\nPooled rate: 660/20,000 = 3.3%\nSE = sqrt(0.033 × 0.967 × 2/10,000) = 0.00253\nz = 0.006/0.00253 = 2.37\np = 0.018 < 0.05
Result:Significant at 95% confidence | 20% lift | p = 0.018
Example 2: Button Color Test (Not Significant)
Problem:Control: 2,000 visitors, 50 conversions (2.5%). Variant: 2,000 visitors, 55 conversions (2.75%). Is this significant?
Solution:Lift: (2.75-2.5)/2.5 = 10.0%\nPooled rate: 105/4,000 = 2.625%\nSE = sqrt(0.02625 × 0.97375 × 2/2,000) = 0.00504\nz = 0.0025/0.00504 = 0.496\np = 0.62 > 0.05
Result:Not significant (p = 0.62) — need more data
Abtest Significance Calculator — Frequently Asked Questions
What is A/B test statistical significance?
Statistical significance in A/B testing means the observed difference between control and variant is unlikely to be due to random chance alone. The standard threshold is p < 0.05 (95% confidence), meaning there is less than a 5% probability the result occurred by chance. A significant result does not mean the effect is large or practically important — it only means it is unlikely to be zero.