A B Test Significance Power Analyzer

This tool runs a two-proportion z-test. Each arm's conversion rate is a binomial proportion; under the null hypothesis of no difference the two arms share a pooled rate, and the pooled standard error is the square root of p_pool times one minus p_pool times the sum of the reciprocals of the two sample sizes. The observed difference divided by that standard error gives a z score, and the two-sided p-value is twice the normal tail area beyond its absolute value. The confidence interval reported for the difference uses the unpooled standard error instead, because once you are estimating the size of an effect rather than testing whether it is zero, the two arms should no longer be assumed to share a common rate. Significance and power answer different questions. Alpha, set here by the 90, 95, or 99 percent confidence choice, caps the false positive rate at 0.10, 0.05, or 0.01. Power is the probability of detecting a real effect when one exists, and 80 percent is the conventional floor, which is where the z value of 0.842 for beta in the sample size formula comes from. Two caveats dominate practice. Post-hoc power computed from the observed effect is largely a restatement of the p-value and should not be used to interpret a null result. And the normal approximation degrades when expected counts per cell drop below roughly 5 to 10, which happens quickly at low conversion rates. Checking results repeatedly and stopping at the first significant reading inflates the true false positive rate well above the nominal alpha.

The machinery here is classical frequentist statistics assembled between 1908 and 1935. William Sealy Gosset, working as a brewer at Guinness in Dublin, published the t distribution in 1908 under the pen name Student because his employer barred staff from publishing under their own names. Ronald A. Fisher, at Rothamsted Experimental Station, formalized randomized experimental design, significance testing, and the analysis of variance in Statistical Methods for Research Workers in 1925 and The Design of Experiments in 1935, and it is Fisher's casual suggestion of a one in twenty threshold that hardened into the 0.05 convention still used as the default here. Jerzy Neyman and Egon Pearson supplied the complementary framework in papers published in 1928 and 1933, introducing the alternative hypothesis, Type I and Type II errors, and the concept of power that this calculator reports. Online controlled experiments then applied that apparatus to web traffic at a scale the original authors never contemplated. Google ran its first live search experiment in 2000, Amazon and Microsoft built internal experimentation platforms over the following decade, and by the 2010s continuous split testing was routine at large internet firms. Ron Kohavi and colleagues documented the operational pitfalls, including the peeking problem and the gap between statistical and practical significance, in a long series of papers and in the 2020 book Trustworthy Online Controlled Experiments.