Q: How many visitors do I need for an A/B test?

Sample size depends on your baseline conversion rate, the minimum detectable effect (MDE) you care about, and the desired statistical power (typically 80%). For a baseline of 3% conversion and a 20% relative lift (3% to 3.6%), you need roughly 15,000-20,000 visitors per variation. Smaller effects require larger samples. Use a power calculator before starting your test to avoid underpowered experiments.

Q: What is the difference between p-value and confidence level?

The p-value is the probability of observing results as extreme as (or more extreme than) the actual results, assuming no real difference exists. The confidence level is 1 - p-value (expressed as a percentage). A p-value of 0.03 means 97% confidence. The convention is to require p < 0.05 (95% confidence) to declare significance, but 99% confidence (p < 0.01) is used for high-stakes decisions.

Q: What does lift mean in A/B testing?

Lift is the relative improvement of the variant over the control. It is calculated as: (Variant Rate - Control Rate) / Control Rate x 100. For example, if the control converts at 3.0% and the variant at 3.8%, the lift is (3.8-3.0)/3.0 = 26.7%. Lift tells you the practical magnitude of the improvement. A statistically significant result with very low lift (e.g., 1-2%) may not be worth implementing.

Q: Why is the pooled proportion used in the standard error instead of each group's own rate?

The z-test evaluates the null hypothesis that both arms share one true conversion rate. Under that assumption the best estimate of that common rate is the pooled proportion p = (x1 + x2) / (n1 + n2), so the standard error sqrt(p(1-p)(1/n1 + 1/n2)) describes the noise you would see if the null were true. Using separate rates gives the unpooled standard error, which is the right choice for building a confidence interval around the observed difference rather than for testing it.

Q: What is peeking, and how does it inflate false positives?

Peeking means checking significance repeatedly while data accumulates and stopping the moment p drops below 0.05. The fixed-horizon z-test assumes a single test at one predetermined sample size; each extra look is another chance for random noise to cross the threshold, so the real Type I error rate climbs well above the nominal 5 percent. Fixes are to commit to a sample size in advance, or to use sequential or always-valid methods designed for continuous monitoring.

Q: What is a sample ratio mismatch and why should it stop the analysis?

Sample ratio mismatch is when the actual split between control and variant deviates from the intended allocation by more than chance allows — for example 52/48 traffic on a nominal 50/50 test with hundreds of thousands of visitors. It signals a broken randomizer, bot filtering that hits one arm harder, or redirect latency dropping users. Because the arms are then no longer comparable, the p-value is meaningless; diagnose the bucketing before interpreting any lift.

Q: When should I use a one-tailed test instead of a two-tailed test?

Abtest Significance Calculator reports a two-tailed p-value, which tests for a difference in either direction and is the safe default. A one-tailed test concentrates the whole rejection region on one side, so it reaches significance sooner but is blind to the variant performing worse. It is defensible only when a decrease is genuinely irrelevant to the decision and the direction was fixed before data collection. Switching to one-tailed after seeing the results is a form of p-hacking.

Q: What is the difference between absolute lift and relative lift?

Absolute lift is the raw gap in percentage points: 3.6 percent minus 3.0 percent equals 0.6 points. Relative lift divides that gap by the control rate: 0.6 / 3.0 equals 20 percent. Relative figures sound dramatic on low baselines, so a headline '30 percent lift' on a 0.2 percent conversion rate is only 0.06 points of extra conversion. Forecast revenue from the absolute difference and its confidence interval, not from the relative number.

Q: Does the normal approximation hold when conversions are rare?

The z-test relies on the binomial distribution being well approximated by a normal curve, which requires enough successes and failures in each arm — a common working rule is at least 10 expected conversions and 10 expected non-conversions per group. With very rare events, such as a handful of purchases on a few thousand sessions, the approximation is unstable and the p-value unreliable. Fisher's exact test or an exact binomial approach is the better tool there.

Question 1

What is A/B test statistical significance?

Accepted Answer

Statistical significance in A/B testing means the observed difference between control and variant is unlikely to be due to random chance alone. The standard threshold is p < 0.05 (95% confidence), meaning there is less than a 5% probability the result occurred by chance. A significant result does not mean the effect is large or practically important — it only means it is unlikely to be zero.

Question 2

How many visitors do I need for an A/B test?

Accepted Answer

Sample size depends on your baseline conversion rate, the minimum detectable effect (MDE) you care about, and the desired statistical power (typically 80%). For a baseline of 3% conversion and a 20% relative lift (3% to 3.6%), you need roughly 15,000-20,000 visitors per variation. Smaller effects require larger samples. Use a power calculator before starting your test to avoid underpowered experiments.

Question 3

What is the difference between p-value and confidence level?

Accepted Answer

The p-value is the probability of observing results as extreme as (or more extreme than) the actual results, assuming no real difference exists. The confidence level is 1 - p-value (expressed as a percentage). A p-value of 0.03 means 97% confidence. The convention is to require p < 0.05 (95% confidence) to declare significance, but 99% confidence (p < 0.01) is used for high-stakes decisions.

Question 4

What does lift mean in A/B testing?

Accepted Answer

Lift is the relative improvement of the variant over the control. It is calculated as: (Variant Rate - Control Rate) / Control Rate x 100. For example, if the control converts at 3.0% and the variant at 3.8%, the lift is (3.8-3.0)/3.0 = 26.7%. Lift tells you the practical magnitude of the improvement. A statistically significant result with very low lift (e.g., 1-2%) may not be worth implementing.

Question 5

Why is the pooled proportion used in the standard error instead of each group's own rate?

Accepted Answer

The z-test evaluates the null hypothesis that both arms share one true conversion rate. Under that assumption the best estimate of that common rate is the pooled proportion p = (x1 + x2) / (n1 + n2), so the standard error sqrt(p(1-p)(1/n1 + 1/n2)) describes the noise you would see if the null were true. Using separate rates gives the unpooled standard error, which is the right choice for building a confidence interval around the observed difference rather than for testing it.

Question 6

What is peeking, and how does it inflate false positives?

Accepted Answer

Peeking means checking significance repeatedly while data accumulates and stopping the moment p drops below 0.05. The fixed-horizon z-test assumes a single test at one predetermined sample size; each extra look is another chance for random noise to cross the threshold, so the real Type I error rate climbs well above the nominal 5 percent. Fixes are to commit to a sample size in advance, or to use sequential or always-valid methods designed for continuous monitoring.

Question 7

What is a sample ratio mismatch and why should it stop the analysis?

Accepted Answer

Sample ratio mismatch is when the actual split between control and variant deviates from the intended allocation by more than chance allows — for example 52/48 traffic on a nominal 50/50 test with hundreds of thousands of visitors. It signals a broken randomizer, bot filtering that hits one arm harder, or redirect latency dropping users. Because the arms are then no longer comparable, the p-value is meaningless; diagnose the bucketing before interpreting any lift.

Question 8

When should I use a one-tailed test instead of a two-tailed test?

Accepted Answer

Abtest Significance Calculator reports a two-tailed p-value, which tests for a difference in either direction and is the safe default. A one-tailed test concentrates the whole rejection region on one side, so it reaches significance sooner but is blind to the variant performing worse. It is defensible only when a decrease is genuinely irrelevant to the decision and the direction was fixed before data collection. Switching to one-tailed after seeing the results is a form of p-hacking.

Question 9

What is the difference between absolute lift and relative lift?

Accepted Answer

Absolute lift is the raw gap in percentage points: 3.6 percent minus 3.0 percent equals 0.6 points. Relative lift divides that gap by the control rate: 0.6 / 3.0 equals 20 percent. Relative figures sound dramatic on low baselines, so a headline '30 percent lift' on a 0.2 percent conversion rate is only 0.06 points of extra conversion. Forecast revenue from the absolute difference and its confidence interval, not from the relative number.

Question 10

Does the normal approximation hold when conversions are rare?

Accepted Answer

The z-test relies on the binomial distribution being well approximated by a normal curve, which requires enough successes and failures in each arm — a common working rule is at least 10 expected conversions and 10 expected non-conversions per group. With very rare events, such as a handful of purchases on a few thousand sessions, the approximation is unstable and the p-value unreliable. Fisher's exact test or an exact binomial approach is the better tool there.

Abtest Significance Calculator