A B Test Significance Power Analyzer
Use our free Test significance power tool to get instant, accurate results. Powered by proven algorithms with clear explanations.
Calculator
Adjust values & calculateDetailed Statistics
Formula
The z-test for two proportions compares conversion rates by computing the pooled standard error and testing whether the observed difference is larger than expected by chance. The p-value from the z-score determines statistical significance.
Last reviewed: December 2025
Worked Examples
Example 1: E-commerce Button Color Test
Example 2: Landing Page Headline Test
Background & Theory
The A B Test Significance Power Analyzer applies the following established principles and formulas. Large language models process text by breaking it into tokens, sub-word units produced by algorithms such as byte-pair encoding. In English, one token approximates four characters or three-quarters of a word on average, though this ratio varies considerably across languages and code. A 1000-word document typically requires around 1300 to 1500 tokens. Token count drives both context window constraints and inference billing, making accurate estimation essential for budgeting API usage. The capability of a neural network scales primarily with its parameter count. Parameters are the numerical weights adjusted during training via gradient descent. GPT-3 contains 175 billion parameters; larger models in the trillion-parameter range require correspondingly greater compute and memory. Training compute is measured in floating-point operations (FLOPs): the Chinchilla scaling laws derived by Hoffmann et al. in 2022 show that optimal training allocates roughly 20 tokens per parameter, meaning a 70B-parameter model benefits from approximately 1.4 trillion training tokens. Inference latency depends on model size, hardware, and batching strategy. Running a 7B-parameter model in FP16 precision requires roughly 14 GB of GPU VRAM (2 bytes per parameter), while INT8 quantisation halves this to around 7 GB with modest quality loss, and INT4 reduces it to approximately 3.5 GB. This quantisation trade-off between memory, speed, and accuracy is central to deploying models on consumer hardware. Perplexity measures how surprised a language model is by a given text corpus; lower perplexity indicates better predictive accuracy. Embedding dimensions determine the size of the dense vector representations used to encode semantic meaning. Models like OpenAI's text-embedding-ada-002 produce 1536-dimensional vectors, while compact models may use 384 dimensions. Context window size defines the maximum token span a model can attend to in a single forward pass. Extending context windows from 4K to 128K tokens enables document-scale reasoning but substantially increases memory requirements, as the attention mechanism scales quadratically with sequence length without architectural modifications such as flash attention.
History
The history behind the A B Test Significance Power Analyzer traces back through the following developments. The mathematical neuron model published by Warren McCulloch and Walter Pitts in 1943 first proposed that logical functions could be computed by networks of simple threshold units, planting the seed of neural computation. Frank Rosenblatt's Perceptron, introduced in 1957 and implemented in custom hardware by 1960, could learn linear classifiers from examples and generated enormous public excitement before Marvin Minsky and Seymour Papert's 1969 book rigorously analysed its fundamental limitations, demonstrating it could not learn the simple XOR function. The first AI winter, roughly 1974 to 1980, followed as funding agencies in the US and UK grew disillusioned with unrealised promises. A second wave of interest during the 1980s produced rule-based expert systems deployed in medicine and finance, and saw the re-derivation of backpropagation by Rumelhart, Hinton, and Williams in 1986, making it practical to train multi-layer networks on real problems. A second winter from 1987 to 1993 followed as expert systems proved brittle and hardware remained insufficient for genuine deep learning. The deep learning revival crystallised at the ImageNet Large Scale Visual Recognition Challenge in 2012, when Alex Krizhevsky's convolutional network AlexNet slashed the top-5 error rate by nearly 11 percentage points compared to the prior year's winner. This demonstrated that deep networks trained on GPUs with large labelled datasets could achieve human-competitive image recognition. Subsequent years saw rapid advances in recurrent networks, sequence-to-sequence models, and the attention mechanism, culminating in the transformer architecture introduced by Vaswani et al. in 2017. OpenAI released GPT-1 in 2018, demonstrating that unsupervised pre-training on large text corpora followed by task-specific fine-tuning could transfer knowledge broadly across language tasks. GPT-2 in 2019 demonstrated surprisingly fluent long-form text generation. GPT-3 in 2020, with 175 billion parameters, showed that scale alone could unlock few-shot learning. Kaplan et al.'s 2020 scaling laws paper provided the theoretical grounding. ChatGPT launched in November 2022, reaching one million users within five days and igniting mainstream global awareness of large language models.
Frequently Asked Questions
Formula
Z = (pB - pA) / sqrt(p_pool x (1 - p_pool) x (1/nA + 1/nB))
The z-test for two proportions compares conversion rates by computing the pooled standard error and testing whether the observed difference is larger than expected by chance. The p-value from the z-score determines statistical significance.
Worked Examples
Example 1: E-commerce Button Color Test
Problem: Test a new green checkout button (B) against the original blue (A). Control: 10,000 visitors, 320 conversions. Variant: 10,000 visitors, 380 conversions. Use 95% confidence.
Solution: Rate A: 320/10000 = 3.200%\nRate B: 380/10000 = 3.800%\nPooled rate: 700/20000 = 3.500%\nSE = sqrt(0.035 x 0.965 x (1/10000 + 1/10000)) = 0.002603\nZ = (0.038 - 0.032) / 0.002603 = 2.305\np-value = 0.0212\nSince p < 0.05, the result is statistically significant.
Result: Significant at 95% | Lift: +18.75% | p-value: 0.021 | Variant B wins
Example 2: Landing Page Headline Test
Problem: Test a new headline (B) vs original (A). Control: 3,000 visitors, 90 signups. Variant: 3,000 visitors, 105 signups. 95% confidence level.
Solution: Rate A: 90/3000 = 3.000%\nRate B: 105/3000 = 3.500%\nPooled rate: 195/6000 = 3.250%\nSE = sqrt(0.0325 x 0.9675 x (1/3000 + 1/3000)) = 0.004575\nZ = (0.035 - 0.030) / 0.004575 = 1.093\np-value = 0.2745\nSince p > 0.05, the result is NOT statistically significant.
Result: Not Significant | Lift: +16.67% | p-value: 0.275 | Need ~14,000 visitors per group for 80% power
Frequently Asked Questions
What is statistical significance in A/B testing?
Statistical significance in A/B testing tells you whether the observed difference between your control (A) and variant (B) is likely due to a real effect rather than random chance. It is quantified by the p-value, which represents the probability of observing a difference as large as (or larger than) what you measured, assuming there is actually no real difference between the two versions. A commonly used threshold is p < 0.05, meaning there is less than a 5% chance the result is due to random variation. However, statistical significance alone does not tell you the practical importance of the difference or whether the observed lift is meaningful for your business. You should always consider effect size and confidence intervals alongside significance.
What is statistical power and why does it matter?
Statistical power is the probability that your test will correctly detect a real difference between variants when one actually exists. A test with 80% power has an 80% chance of detecting a true effect and a 20% chance of missing it (a Type II error or false negative). Power depends on four factors: sample size, effect size (how big the real difference is), significance level (alpha), and baseline conversion rate. Running underpowered tests is a common mistake that leads teams to conclude that a variant has no effect when it actually does. Before starting an A/B test, you should calculate the required sample size to achieve at least 80% power for the minimum detectable effect that would be practically meaningful to your business.
How long should I run an A/B test?
You should run an A/B test until you reach the pre-calculated sample size needed for adequate statistical power, typically 80% or higher. Stopping a test early because it looks significant (called peeking) inflates your false positive rate dramatically. As a guideline, most tests should run for at least one full business cycle (usually one to two weeks) to account for day-of-week effects and traffic patterns. Additionally, never run a test indefinitely hoping for significance, as this is a form of p-hacking. If your traffic is low, you may need to test larger changes that produce bigger effect sizes, or accept that you need several weeks or months of data. Tools like sequential testing or Bayesian methods can allow valid early stopping.
Can I use the results for professional or academic purposes?
You may use the results for reference and educational purposes. For professional reports, academic papers, or critical decisions, we recommend verifying outputs against peer-reviewed sources or consulting a qualified expert in the relevant field.
What inputs do I need to use A B Test Significance Power Analyzer accurately?
Each field is labelled with the required unit (metric or imperial). Gather your source values before starting — for example, a weight measurement in kilograms, a distance in metres, or a dollar amount — and enter them exactly as measured. The formula section on this page lists every variable and explains what each represents.
Does A B Test Significance Power Analyzer work offline?
Once the page is loaded, the calculation logic runs entirely in your browser. If you have already opened the page, most calculators will continue to work even if your internet connection is lost, since no server requests are needed for computation.
References
Reviewed by Daniel Agrici, Founder & Lead Developer · Editorial policy