CUPED Variance Reduction

Worked Examples

Example 1: E-commerce Revenue Experiment

Problem: An e-commerce site runs a checkout flow A/B test. Revenue per user has high variance (σ²=2500). Pre-experiment revenue correlation is 0.65. Current sample: 50,000 per variant.

Solution: CUPED Analysis:\nCorrelation (r) = 0.65\nBaseline variance = 2500\nSample size = 50,000 per variant\n\nVariance Reduction:\nr² = 0.65² = 0.4225\nReduction = 42.25%\n\nNew Variance:\n2500 × (1 - 0.4225) = 1444\n\nStandard Error Impact:\nBaseline SE = √(2500/50000) = 0.224\nCUPED SE = √(1444/50000) = 0.170\nSE reduction = 24%\n\nMDE Impact (80% power, 95% CI):\nBaseline MDE = 2.8 × √2 × 0.224 = $0.89\nCUPED MDE = 2.8 × √2 × 0.170 = $0.67\nCan now detect 25% smaller effects!\n\nRuntime Equivalent:\n50,000 with CUPED ≈ 87,000 without\nSaves 37,000 users worth of time

Result: 42% variance reduction | 25% smaller MDE | Equivalent to 87,000 users/variant

Example 2: Engagement Metric Optimization

Problem: A social app tests a new feed algorithm. Sessions per user is the metric. Pre-experiment correlation is only 0.4. Is CUPED worth implementing?

Solution: Correlation Analysis:\nr = 0.4\nr² = 0.16\nVariance reduction = 16%\n\nRuntime Impact:\nRuntime reduction ≈ 1 - √(1-0.16)\n= 1 - 0.917 = 8.3%\n\nA/B test running 4 weeks:\n4 weeks × 8.3% = 0.33 weeks = 2.3 days saved\n\nImplementation Trade-off:\n- Engineering effort: ~1-2 weeks\n- Runtime savings: 2 days per test\n- Break-even: ~5-7 experiments\n\nImproving Correlation:\n- Use 2-week pre-period instead of 1-week\n- Combine multiple pre-period metrics\n- Segment by user tenure (new vs returning)\n\nIf improved to r=0.55:\nr² = 0.30\nRuntime reduction ≈ 16%\nSaves ~4.5 days on 4-week test\nMuch better ROI

Result: 16% variance reduction (r=0.4) | 8% faster | Consider improving covariate first

Example 3: Netflix-Style CUPED Implementation

Problem: A streaming service runs 100+ A/B tests annually on watch time. Pre-experiment watch time correlation is 0.75. Calculate the platform-wide value of implementing CUPED.

Solution: Single Experiment Analysis:\nr = 0.75\nr² = 0.5625\nVariance reduction = 56%\nRuntime reduction = 34%\n\nTypical experiment: 2 weeks\nWith CUPED: 2 × (1-0.34) = 1.32 weeks\nSavings per test: 4.8 days\n\nPlatform-Wide Impact:\n100 experiments/year × 4.8 days = 480 days\n= 16 experiment-months saved\n\nAlternative view - throughput increase:\nPreviously: 100 tests in 52 weeks\nWith CUPED: Can run 152 tests in same time\n52% more experiments!\n\nPower Improvement:\nIf experiments were 80% powered:\nSame sample now = 95%+ power\nOr: detect 25% smaller effects\n\nBusiness Value:\nFaster learning → faster shipping\n52% more tests → faster iteration\nHigher power → fewer false negatives\nCompetitive advantage: significant

Result: 56% variance reduction | 52% more tests annually | 480 experiment-days saved/year

Frequently Asked Questions

What is CUPED?

CUPED (Controlled-experiment Using Pre-Experiment Data) is a variance reduction technique that uses pre-experiment user behavior to reduce noise in A/B test metrics. By controlling for pre-existing differences, CUPED increases statistical power without needing more users.

How does CUPED reduce variance?

CUPED adjusts each user's metric by their pre-experiment behavior. If a user historically has high engagement, we expect high engagement during the experiment. Subtracting this expected value removes predictable variation, leaving only the experiment's true effect plus random noise.

What correlation do I need for CUPED to work?

Variance reduction equals correlation squared (r²). A 0.5 correlation gives 25% variance reduction; 0.7 gives 49%; 0.8 gives 64%. Correlations below 0.3 provide minimal benefit (<9%). Most web metrics achieve 0.4-0.7 correlation with pre-period data.

Does CUPED change my experiment results?

CUPED doesn't bias results - it reduces variance without changing the expected treatment effect. The adjusted metric has the same mean difference between variants but less noise. This means higher confidence in whatever effect you observe.

Can CUPED make my experiment faster?

Yes! Variance reduction translates to faster experiments. With 50% variance reduction, you reach the same statistical power in roughly 50% of the time. Many companies report 30-50% experiment runtime reductions with CUPED.

What's the formula for CUPED adjustment?

Y_adjusted = Y - θ(X - X̄), where Y is the experiment metric, X is the pre-experiment covariate, and θ = Cov(X,Y)/Var(X). This regression-based adjustment removes the portion of Y variance explained by X.

Background & Theory

The Variance Reduction CUPED Estimator applies the following established principles and formulas. Statistics and probability provide the mathematical framework for drawing conclusions from data under uncertainty. The measures of central tendency describe where data cluster. The mean is the arithmetic average, computed as the sum of all values divided by the count. The median is the middle value of an ordered dataset, robust to extreme outliers. The mode is the most frequent value. Spread is quantified by variance, the average squared deviation from the mean, and by its square root, the standard deviation. For a sample, variance uses n minus one in the denominator to correct for bias in estimation. The normal distribution, defined by its mean and standard deviation, is the cornerstone of parametric statistics. Its bell-shaped probability density follows the formula f(x) = (1 / (sigma * sqrt(2*pi))) * exp(-0.5 * ((x - mu) / sigma)^2). The empirical rule states that approximately 68 percent of observations fall within one standard deviation of the mean, 95 percent within two, and 99.7 percent within three. A z-score standardizes a data point by subtracting the mean and dividing by the standard deviation, expressing how many standard deviations an observation lies from the mean. In hypothesis testing, the p-value is the probability of observing a result at least as extreme as the one obtained, assuming the null hypothesis is true. Confidence intervals express the range within which the true population parameter falls with a specified probability, typically 95 percent. Correlation measures linear association between two variables, with Pearson's r ranging from negative one to positive one. Correlation does not imply causation. Linear regression fits a line of the form y = a + bx to minimize the sum of squared residuals. Bayes' theorem relates conditional probabilities: P(A|B) = P(B|A) * P(A) / P(B), allowing prior beliefs to be updated on new evidence. The law of large numbers guarantees that the sample mean converges to the population mean as sample size grows. The central limit theorem states that the distribution of sample means approaches normality regardless of the population distribution, provided the sample size is sufficiently large, typically 30 or more.

History

The history behind the Variance Reduction CUPED Estimator traces back through the following developments. The mathematical study of probability emerged in the 17th century from correspondence between Blaise Pascal and Pierre de Fermat in 1654. Their exchange, prompted by a gambling problem posed by the Chevalier de Mere, established the foundations of probability theory by calculating expected outcomes through systematic enumeration of cases. Jacob Bernoulli formalized the law of large numbers in his posthumously published Ars Conjectandi of 1713, proving rigorously that empirical frequencies converge to theoretical probabilities with increasing observations. His work laid the groundwork for inferential statistics by connecting mathematical probability to observed data. Carl Friedrich Gauss developed the method of least squares around 1795 while adjusting astronomical observations, and he recognized the bell-shaped error distribution that now bears his name. Pierre-Simon Laplace independently worked on the normal distribution and proved an early version of the central limit theorem around 1810, demonstrating why errors in measurement tend toward normality. The late 19th century saw statistics emerge as a distinct scientific discipline. Francis Galton introduced regression and correlation in the 1880s while studying heredity. Karl Pearson formalized these concepts, developed the chi-squared test, and founded the journal Biometrika in 1901, establishing statistics as a rigorous academic field. Ronald Fisher transformed statistical practice in the early 20th century. His 1925 book Statistical Methods for Research Workers introduced significance testing, analysis of variance, and the concept of the p-value as a decision threshold, establishing the framework still used in scientific research. Fisher and Jerzy Neyman engaged in a prolonged methodological dispute over the interpretation of hypothesis tests. The Bayesian approach, rooted in the 18th century work of Thomas Bayes and Laplace, was largely eclipsed by frequentist methods through much of the 20th century but experienced a revival after World War II and accelerated with computational advances. The late 20th and early 21st centuries brought statistics into every domain through big data, machine learning, and the routine availability of software capable of processing millions of observations.