Incrementality Lift Planner
Plan marketing holdout tests to measure true incremental lift. Calculate the revenue impact of your ads compared to a control group that didn't see them.
Formula
IncrementalRevenue = (TestConv% - ControlConv%) × TestAudienceSize × AOV
We compare the behavior of users who saw ads (Test) vs. a randomized group who didn't (Control/Holdout). The difference in their conversion rates represents the 'Lift' generated specifically by the advertising, stripping out organic demand.
Worked Examples
Example 1: Successful Campaign
Problem:Test Conv 5.5%, Control Conv 5.0%. Test Size 90k.
Solution:Diff = 0.5%. 90,000 * 0.005 = 450 Incremental Orders.
Result:+10% Relative Lift
Example 2: Cannibalization
Problem:Test Conv 5.0%, Control Conv 5.0%.
Solution:Diff = 0%. Ads just paid for conversions that would have happened anyway.
Result:0% Lift (Wasted Spend)
Frequently Asked Questions
What is Incrementality?
The measure of the true causal impact of marketing. It answers: 'Would these customers have bought anyway if I hadn't shown them an ad?'
What is a Holdout Group?
A portion of your audience (usually 10-20%) that is deliberately excluded from seeing your ads. They serve as the baseline (Control) to measure effectiveness.
Why not just use ROAS?
ROAS (Return on Ad Spend) is often inflated by attribution models taking credit for organic sales. iROAS (Incremental ROAS) is the truth.
What is a Ghost Ad?
A technique where you identify users who *would* have seen your ad in the control group, but show them a PSA or nothing instead, to ensure fair comparison.
How large should the holdout be?
Typically 10-20%. Too small, and you lack statistical significance. Too large, and you lose potential revenue from the campaign.
Can I do this on Facebook/Google?
Yes, they offer 'Conversion Lift' studies (Brand Lift/Conversion Lift) which automate the holdout groups.
What is 'Cannibalization'?
When paid ads target users who were already going to buy via organic search or direct traffic. You pay to acquire a customer you already had.
Is this A/B testing?
It's a form of A/B testing (A=Ad, B=No Ad), specifically designed to measure media effectiveness rather than creative performance.
How long should a test run?
At least one full business cycle (usually 2-4 weeks) to account for day-of-week fluctuations and attribution windows.
What is Relative Lift?
The percentage increase over the baseline. If baseline is 2% and test is 3%, the absolute lift is 1% but relative lift is 50%.
Background & Theory
The Causality Problem
Correlation does not equal Causation. Just because someone saw an ad and bought, doesn't mean the ad *caused* the purchase.
Test Design
- Randomization: Groups must be identical.
- Isolation: Ensure the control group isn't exposed via another channel (contamination).
- Significance: Need enough sample size to detect small lifts (e.g., 5% lift on a 2% base).
Interpretation Guide
- High Lift, Low ROAS: Effective at driving new behavior, but expensive. Good for growth.
- Low Lift, High ROAS: Efficient harvesting of demand, but not generating new demand. Good for profitability, but caps growth.
History
The Click-Through Era
In the 1990s, we measured clicks. If someone clicked, it worked. Simple.
The Attribution Wars
In the 2000s/2010s, we tracked conversions. But "Last Click" attribution gave 100% credit to the last ad (often a branded search term). Marketers spent millions bidding on their own brand name, claiming 50x ROAS, while organic traffic plummeted.
The Experimentation Era
Sophisticated advertisers (eBay, Uber, Netflix) paused their spend and noticed... nothing happened. Sales didn't drop. This "Blackout" revealed the lie of attribution. Today, Incrementality Testing (Geo-lift, User-lift) is the gold standard for verifying that ad spend actually drives *net new* revenue.
Common Misconceptions
- Myth: High ROAS = Good Campaign. Reality: High ROAS often means you are retargeting loyal customers who were buying anyway.
- Myth: Holdouts lose money. Reality: They save money by exposing ineffective spend.