Incrementality Testing: The Only Way to Know If Your Ads Actually Work
Stop guessing if your ads work. Incrementality testing reveals true ad impact with causal inference, not cookies. Learn how to measure lift accurately in a cookieless world.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 5 min read
Run the numbers for your store: the free holdout test planner.
Incrementality Testing: The Only Way to Know If Your Ads Actually Work
Last-click attribution, multi-touch models, and even the shiny new AI-powered dashboards all share one fatal flaw: they measure correlation, not causation. Incrementality testing is the only method that tells you if your ads actually drive sales. Here’s why you can’t afford to ignore it.
What Is Incrementality Testing and Why Does It Matter
Incrementality testing measures the causal impact of your ads by comparing two identical groups: one exposed to your campaign (test) and one shielded from it (control). The difference in outcomes, sales, sign-ups, revenue, is your incremental lift. No guesswork. No attribution models inflating numbers. Just the cold, hard truth.
The industry standard for measurement is broken. Meanwhile, marketers cling to last-click models that credit 100% of a sale to the final touchpoint. This isn’t just inaccurate: it’s actively misleading.
Incrementality testing fixes this by answering the only question that matters: Would this customer have converted anyway?
Why Cookies and Traditional Attribution Fail
Cookies are dead. Yet most attribution tools still rely on these crumbs to stitch together customer journeys. The result?
- Linear: Spreads credit evenly, ignoring that some touchpoints drive 10x more lift than others.
- Time-decay: Assumes recent interactions matter most, even when brand awareness from a 30-day-old ad drives the final click.
These models don’t measure impact. They measure proximity. And proximity is not causation.
How Causal Inference Solves the Cookieless Measurement Problem
Causal inference doesn’t need cookies. It doesn’t need user-level tracking. It doesn’t even need a customer journey.
1. Randomized Control Trials (RCTs): The Gold Standard
RCTs are the backbone of incrementality testing. You randomly split your audience into test and control groups, expose the test group to your ad, and compare outcomes. No assumptions. No models. Just math.
2. Geo-Experiments: When RCTs Aren’t Feasible
Not every brand can run RCTs. For large-scale campaigns, geo-experiments compare outcomes in regions exposed to ads vs. those held out.
3. Synthetic Control Methods: The Cookieless Workaround
Synthetic control methods create a “digital twin” of your test group using pre-campaign data. This twin acts as the control, letting you measure lift without user-level tracking.
How to Run an Incrementality Test: A Step-by-Step Guide
Step 1: Define Your Hypothesis
What do you want to test? Be specific. Not “Do my ads work?” but “Does retargeting on Facebook drive incremental purchases among lapsed customers?”
Step 2: Choose Your Method
- RCTs: Best for digital campaigns with precise audience targeting.
- Geo-experiments: Best for offline or large-scale digital campaigns.
- Synthetic controls: Best for cookieless environments or when RCTs aren’t feasible.
Step 3: Set Up Your Test and Control Groups
- Geo-experiments: Select regions with similar pre-campaign performance. Avoid overlapping DMAs.
- Synthetic controls: Use historical data to model the control group’s expected behavior.
Step 4: Run the Test
- Budget: Allocate enough spend to detect lift.
Step 5: Analyze the Results
Calculate incremental lift using this formula:
Incremental Lift = (Test Group Conversion Rate - Control Group Conversion Rate) / Control Group Conversion Rate
Example: If your test group converts at 5% and your control at 3%, your incremental lift is (5% - 3%) / 3% = 66.7%.
Step 6: Refine Based on Findings
- Double down on high-incrementality channels.
- Cut or retool low-incrementality tactics.
- Test again to validate changes.
The Incrementality Testing Playbook: What to Test First
Not all tests are created equal. Start with these high-impact experiments:
1. Retargeting vs. Prospecting
Hypothesis: Retargeting ads drive incremental conversions among cart abandoners.
2. Brand vs. Non-Brand Search
Hypothesis: Non-brand search ads drive incremental conversions.
Shift budget from brand to non-brand or use brand search as a defensive tactic.
3. Social Ads vs. Influencer Marketing
Hypothesis: Influencer marketing drives incremental sales.
4. Email Frequency
Hypothesis: Sending more emails increases incremental revenue.
Use incrementality testing to find your brand’s sweet spot.
How to Get Started with Incrementality Testing
You don’t need a PhD in statistics to run incrementality tests. Here’s how to begin:
1. Start Small
Pick one high-spend channel (e.g., Facebook ads) and run a simple RCT. Use platform tools like Facebook’s Holdout Test or Google’s Experiment Groups to get your feet wet.
3. Build a Testing Culture
Incrementality testing isn’t a one-time project: it’s a mindset. Make it part of your quarterly planning. Test hypotheses, analyze results, and refine. Rinse and repeat.
4. Demand Transparency
If your attribution tool can’t explain why it’s crediting a conversion to a specific touchpoint, it’s not measuring incrementality. Ask for the methodology. If they can’t provide it, walk away.
The Future of Measurement Is Incremental
Cookies are dead. Attribution models are broken. The only way to know if your ads actually work is to measure their causal impact. Incrementality testing isn’t just the future of measurement: it’s the only way to survive in a cookieless world.
The choice is yours. But the clock is ticking.
Stop guessing.
Sources and Further Reading
- Harvard Business Review on Marketing Attribution
- McKinsey on Marketing ROI
- Causality Engine Resources
Related Articles
Frequently asked questions
What’s the difference between incrementality testing and attribution modeling?
Attribution models guess which touchpoints deserve credit. Incrementality testing measures the causal impact of ads by comparing test and control groups. Attribution is correlation; incrementality is causation.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Attribution ModelAn Attribution Model defines how credit for conversions is assigned to marketing touchpoints. It dictates how marketing channels receive credit for sales.
- Causal InferenceCausal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than alongside it. It works by comparing what happened with an estimate of what would have happened without the cause.
- Conversion rateConversion Rate is the percentage of website visitors who complete a desired action out of the total number of visitors.
- Customer journeyCustomer journey is the path and sequence of interactions customers have with a website. Customers use multiple devices and channels, making a consistent experience important.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.
- Influencer MarketingInfluencer Marketing uses endorsements and product placements from individuals with dedicated social followings. It uses trusted voices to promote products.
- Marketing AttributionMarketing attribution assigns credit to marketing touchpoints that contribute to a conversion or sale. Causal inference enhances attribution models by identifying true cause-effect relationships.
- Synthetic Control MethodThe Synthetic Control Method estimates the causal effect of an intervention in a single case study. It constructs a 'synthetic' control unit from a weighted average of control units to isolate the intervention's impact.