Incrementality Testing: The Only Way to Know If Your Ads Actually Work

Incrementality testing is the only method to determine if your advertising budget actually generates new sales. Every single one of them. Meta, Google, TikTok. they all claim credit for the same conversions, leaving you with a bloated ROAS that looks great on a dashboard but fails to materialize in your bank account. But when you do, your entire funnel collapses. This is the daily reality for Dutch Shopify brands, and it is a direct result of relying on broken marketing attribution.

Attribution models are designed to do one thing: assign credit. They are scorekeepers, not scientists. They look for correlations, not causes. A last-touch model gives 100 percent of the credit to the final click, ignoring every interaction that came before. A multi-touch model spreads the credit around based on arbitrary rules. None of them answer the only question that actually matters: what would have happened if I had not run this ad?

Attributed revenue is a metric used by ad platforms to show the total value of conversions they claim to have influenced. Unlike actual revenue, this figure is often inflated because multiple platforms take credit for the same sale. For ecommerce brands, relying on attributed revenue leads to misinformed budget allocation and wasted spend on channels that do not drive real growth.

Let’s be direct. The “revenue” number you see in your Meta Ads dashboard is not real. It is a calculated metric based on a model that is fundamentally designed to make Meta look good. The same is true for Google. Your Shopify backend reports one sale, but your ad platforms report three. This is not just a minor discrepancy. it is a systemic flaw that makes intelligent budget allocation impossible.

This is the core problem of correlation-based attribution. It assumes that because a click happened before a sale, the click caused the sale. But what if that customer was already on their way to buy from you? What if they were already a loyal customer who would have purchased anyway? Your retargeting campaign did not create that sale; it just got in the way. The revenue it claims is not incremental. it is cannibalized from other channels or organic intent. You are paying to acquire customers you already had.

For high-growth Dutch ecommerce brands, especially in the beauty and fashion sectors, this problem is magnified. You operate in a competitive space where every marketing euro needs to be justified. You need proof. You need causality.

From Guesswork to Certainty: The Science of Incrementality Testing

See also: Incrementality Testing Tools: Which Platform Gets It Right?

Incrementality testing is a scientific method used to measure the true causal impact of a marketing campaign. Unlike traditional attribution, which only tracks correlations, incrementality testing isolates the effect of your ads by comparing a group that sees them to a group that does not. This reveals the actual lift in conversions directly caused by your marketing spend.

Incrementality testing, at its core, is a scientific experiment. It moves beyond correlation and measures the true causal impact of your marketing spend. Instead of just tracking what happened, it reveals why it happened by isolating the effect of a single variable: your ad campaign. The methodology is simple in principle but useful in practice. It involves splitting your target audience into two identical groups:

1. The Test Group: This group is exposed to your advertising. 2. The Control Group (or Holdout Group): This group is deliberately not shown your advertising.

By comparing the behavior of the test group to the control group, you can measure the incremental lift generated by your ads. This is the difference in conversion rate, revenue, or any other key metric that can be directly and causally attributed to your marketing efforts. If the test group converts at a 5 percent rate and the control group converts at a 3 percent rate, the incremental lift from your campaign is 2 percent. For more on this, see our guide on causal attribution.

The formula for incremental lift is straightforward:

Incremental Lift = (Test Group Conversion Rate) - (Control Group Conversion Rate)

And the number of incremental sales is calculated as:

Incremental Sales = (Incremental Lift) * (Number of Users in Test Group)

This is not a model. It is a measurement. It is the difference between guessing and knowing. While traditional attribution asks “which channel gets the credit?”, incrementality testing asks “did this channel create a new customer?”.

How to Run an Incrementality Test: A Framework for Shopify Brands

Geo-lift testing is a method of incrementality testing that uses geographic regions as test and control groups. Instead of splitting users randomly, you show ads to one region while withholding them from another comparable region. This approach is practical for Shopify brands and provides clean data on the causal impact of ad campaigns by measuring the difference in conversions between the two areas.

Running a proper incrementality test requires a structured approach. For Shopify brands in the Netherlands, a geo-lift test is often the most practical and effective method. Instead of holding out a random subset of users (which can be technically challenging), you can use geographic regions as your test and control groups. This is particularly effective in a market like the Netherlands, where distinct provinces can serve as clean testing grounds.

Here is a framework for designing and executing a geo-lift test:

1. Select Your Test and Control Regions: Identify two or more geographic areas with similar demographic and behavioral profiles. For example, you might designate North Holland as your test region and South Holland as your control region. The key is to ensure the regions are comparable in terms of population, income levels, and past purchasing behavior.

2. Establish a Baseline: Before launching the test, measure the baseline conversion rate and other key metrics in both regions for a set period (e.g., four weeks). This ensures that any pre-existing differences between the regions are accounted for in your final analysis.

3. Launch the Campaign: Run your ad campaign exclusively in the test region. The control region receives no advertising from this specific campaign. It is critical to maintain this separation for the duration of the test to avoid contamination of the results.

4. Measure and Analyze: After a predetermined period (e.g., another four weeks), measure the conversion rates in both the test and control regions. The difference between the two is your incremental lift. You can then calculate the true, incremental ROAS of your campaign. You can use our ROAS calculator to do this.

Incremental ROAS = (Incremental Revenue) / (Ad Spend)

This number is often shockingly different from the platform-reported ROAS. You can now confidently cut the underperforming campaign, even if its vanity metrics look impressive, and scale the one that is actually driving incremental sales.

Running manual incrementality tests is a useful step towards understanding your true marketing performance. But it can be slow, complex, and resource-intensive.

To learn more about the fundamentals, read our guide to causal inference.

FAQ: Incrementality Testing

What is the main difference between incrementality testing and A/B testing?

Incrementality testing measures the causal impact of an ad by comparing a test group to a control group that sees no ad, determining the ad's true value. A/B testing, however, compares two different versions of a creative or landing page to see which performs better. It optimizes content, while incrementality testing validates spend.

How long should I run an incrementality test?

The ideal duration for an incrementality test depends on your sales cycle and conversion volume. For most Dutch Shopify brands, a test duration of 4 to 6 weeks is sufficient to gather statistically significant data. This timeframe is long enough to capture the full impact without letting external factors like seasonality skew the results.

Can I run incrementality tests on all my marketing channels?

Yes, incrementality testing is a versatile method that can be applied to any marketing channel, from Meta and Google to email and influencer campaigns. The methodology remains the same: isolate the channel’s impact by comparing a test group to a control group. This provides a complete picture of which channels drive real growth.

Is ROAS a useless metric?

Incremental ROAS, however, is a vital metric. It reveals the true return on your ad spend and provides a reliable foundation for making profitable budget allocation decisions. You can calculate your waste with our waste calculator.

Your Next Step

Measure Your True ROAS.

Measure your true ROAS