Abtest Significance Calculator

A two-proportion z-test asks a narrow question: if the control and variant conversion rates were truly identical, how often would random assignment alone produce a gap at least as large as the one observed? The pooled proportion p = (x1 + x2) / (n1 + n2) estimates that common rate under the null hypothesis, and the standard error sqrt(p(1-p)(1/n1 + 1/n2)) describes the sampling noise expected around a difference of zero. Dividing the observed gap by that standard error gives the z statistic, compared against the standard normal curve because the binomial distribution converges to a normal shape as sample size grows. The approximation is reliable when both groups have at least 10 expected conversions and 10 expected non-conversions; below that, an exact test such as the Fisher exact test is more appropriate. Three numbers should be read together, not one. Typical ecommerce checkout rates run 1 to 4 percent, so detecting a 10 percent relative lift on a 2 percent baseline needs roughly 30,000 visitors per arm at 80 percent power. The confidence interval on the absolute difference, expressed in percentage points, communicates precision far better than a bare p-value. Common failure modes are checking results daily and stopping at the first significant reading, which can inflate the false positive rate from 5 percent to over 20 percent; sample ratio mismatch caused by broken bucketing; novelty effects in the first days of exposure; and ending a test mid-week when weekday and weekend traffic convert differently.

The mathematics predate the web by a century. Karl Pearson introduced the chi-square goodness-of-fit test in 1900, giving statisticians their first general tool for comparing observed counts against expectation. Ronald A. Fisher, working on agricultural field trials at Rothamsted Experimental Station in the 1920s, formalized randomization and significance testing in Statistical Methods for Research Workers (1925), where he described a probability of 0.05 as a convenient conventional cut-off. Jerzy Neyman and Egon Pearson added the complementary machinery in 1933: an explicit alternative hypothesis, Type I and Type II errors, and statistical power. The two-proportion z-test used here is a normal-approximation descendant of that combined framework. Applying it to web traffic began around 2000, when Google ran an early live experiment splitting search traffic to test how many results to show per page. Amazon and Microsoft built internal experimentation platforms over the following years, and Ronny Kohavi and colleagues published influential practitioner guides on controlled experiments on the web from 2007 onward. Commercial tooling followed: Google Website Optimizer launched in 2007, and Optimizely was founded in 2010. The scale of online testing exposed the peeking problem, which pushed the field toward sequential methods descended from Abraham Wald's sequential probability ratio test of 1945 and toward always-valid p-values.