A B Test Significance Power Analyzer
Use our free Test significance power tool to get instant, accurate results. Powered by proven algorithms with clear explanations.
Formula
Z = (pB - pA) / sqrt(p_pool x (1 - p_pool) x (1/nA + 1/nB))
The z-test for two proportions compares conversion rates by computing the pooled standard error and testing whether the observed difference is larger than expected by chance. The p-value from the z-score determines statistical significance.
Worked Examples
Example 1: E-commerce Button Color Test
Problem: Test a new green checkout button (B) against the original blue (A). Control: 10,000 visitors, 320 conversions. Variant: 10,000 visitors, 380 conversions. Use 95% confidence.
Solution: Rate A: 320/10000 = 3.200%\nRate B: 380/10000 = 3.800%\nPooled rate: 700/20000 = 3.500%\nSE = sqrt(0.035 x 0.965 x (1/10000 + 1/10000)) = 0.002603\nZ = (0.038 - 0.032) / 0.002603 = 2.305\np-value = 0.0212\nSince p < 0.05, the result is statistically significant.
Result: Significant at 95% | Lift: +18.75% | p-value: 0.021 | Variant B wins
Example 2: Landing Page Headline Test
Problem: Test a new headline (B) vs original (A). Control: 3,000 visitors, 90 signups. Variant: 3,000 visitors, 105 signups. 95% confidence level.
Solution: Rate A: 90/3000 = 3.000%\nRate B: 105/3000 = 3.500%\nPooled rate: 195/6000 = 3.250%\nSE = sqrt(0.0325 x 0.9675 x (1/3000 + 1/3000)) = 0.004575\nZ = (0.035 - 0.030) / 0.004575 = 1.093\np-value = 0.2745\nSince p > 0.05, the result is NOT statistically significant.
Result: Not Significant | Lift: +16.67% | p-value: 0.275 | Need ~14,000 visitors per group for 80% power
Frequently Asked Questions
What is statistical significance in A/B testing?
Statistical significance in A/B testing tells you whether the observed difference between your control (A) and variant (B) is likely due to a real effect rather than random chance. It is quantified by the p-value, which represents the probability of observing a difference as large as (or larger than) what you measured, assuming there is actually no real difference between the two versions. A commonly used threshold is p < 0.05, meaning there is less than a 5% chance the result is due to random variation. However, statistical significance alone does not tell you the practical importance of the difference or whether the observed lift is meaningful for your business. You should always consider effect size and confidence intervals alongside significance.
What is statistical power and why does it matter?
Statistical power is the probability that your test will correctly detect a real difference between variants when one actually exists. A test with 80% power has an 80% chance of detecting a true effect and a 20% chance of missing it (a Type II error or false negative). Power depends on four factors: sample size, effect size (how big the real difference is), significance level (alpha), and baseline conversion rate. Running underpowered tests is a common mistake that leads teams to conclude that a variant has no effect when it actually does. Before starting an A/B test, you should calculate the required sample size to achieve at least 80% power for the minimum detectable effect that would be practically meaningful to your business.
How long should I run an A/B test?
You should run an A/B test until you reach the pre-calculated sample size needed for adequate statistical power, typically 80% or higher. Stopping a test early because it looks significant (called peeking) inflates your false positive rate dramatically. As a guideline, most tests should run for at least one full business cycle (usually one to two weeks) to account for day-of-week effects and traffic patterns. Additionally, never run a test indefinitely hoping for significance, as this is a form of p-hacking. If your traffic is low, you may need to test larger changes that produce bigger effect sizes, or accept that you need several weeks or months of data. Tools like sequential testing or Bayesian methods can allow valid early stopping.
Can I use the results for professional or academic purposes?
You may use the results for reference and educational purposes. For professional reports, academic papers, or critical decisions, we recommend verifying outputs against peer-reviewed sources or consulting a qualified expert in the relevant field.
Is A B Test Significance Power Analyzer free to use?
Yes, completely free with no sign-up required. All calculators on NovaCalculator are free to use without registration, subscription, or payment.
Does A B Test Significance Power Analyzer work offline?
Once the page is loaded, the calculation logic runs entirely in your browser. If you have already opened the page, most calculators will continue to work even if your internet connection is lost, since no server requests are needed for computation.