Geo-Lift Testing for Ecommerce: A Practical Guide to Measuring Incrementality

Geo-lift testing provides a clear, scientific method for ecommerce brands to measure the true causal impact of their advertising. It works by dividing a region into test and control groups, running ads only in the test area, and then measuring the difference in sales. This allows you to finally understand the incremental revenue your campaigns are generating, moving beyond the flawed and misleading metrics provided by ad platforms.

The Attribution Illusion: Why Your ROAS is a Vanity Metric

Return on Ad Spend (ROAS) is a metric used by ad platforms to measure the revenue generated for every dollar spent on advertising. Unlike true incremental lift, platform ROAS is a vanity metric based on flawed correlational data. It fails to distinguish between sales that were caused by ads and sales that would have happened anyway, leading to inflated and misleading performance reports.

For years, marketers have been conditioned to chase ROAS. Platforms like Meta and Google plaster it all over their dashboards, creating a useful incentive to tune for this single metric. The problem is, the ROAS you see in your ad manager is a calculated illusion, a number designed to encourage more spending, not to reflect ground truth. These platforms rely on outdated, correlational marketing attribution models like last-touch or multi-touch. These models operate on a series of flawed assumptions:

• A spike in sales might be attributed to your campaign when, in reality, it was caused by a viral TikTok trend. * Correlation is Not Causation: This is the most critical failure. Attribution models are masters at finding correlations. A user saw your ad and then converted. But did the ad cause the conversion? Or was that user already on their way to purchase, and the ad simply got in the way? Traditional attribution cannot tell the difference. It’s like claiming that roosters cause the sun to rise because they crow every morning at dawn.

You are paying to reach customers you already have. You are pouring money into cannibalistic channels that are stealing credit, not creating value. To understand the true performance of your marketing, you need to measure incrementality, a concept we explore in our /blog/what-is-incrementality post.

Geo-Lift Testing: The Scientific Method for Marketers

Geo-lift testing is a controlled experiment that measures the causal effect of an ad campaign by dividing a region into a test group that sees ads and a control group that does not. Unlike user-level A/B testing, geo-lift testing measures the impact on a whole population, providing a true measure of incremental sales caused by marketing efforts.

If attribution is the problem, what is the solution? A geo-lift test, or geo-experiment, is a controlled experiment that measures the causal effect of an ad campaign. Instead of A/B testing creative elements on individual users, you are testing the impact of an entire campaign on a geographic population. It’s the scientific method, applied to marketing.

The methodology is elegant in its simplicity:

1. Divide: Split a country or region into two groups of distinct geographic areas: a Test Group and a Control Group. 2. Isolate: Run your advertising campaign exclusively in the Test Group. The Control Group receives no advertising from this campaign. 3. Measure: Compare the sales results between the two groups. The difference, or "lift," represents the incremental sales caused by your campaign.

This approach bypasses the need for individual user tracking entirely. It is immune to cookie deprecation and privacy updates because it measures aggregate effects at a population level. It provides a direct, causal link between your marketing spend and the incremental revenue it generates.

A Practical Guide to Designing Your First Geo-Lift Test

Designing a geo-lift test involves a structured process of defining a clear hypothesis, selecting and matching geographic markets, and determining the test's duration and budget. Unlike simply running an ad campaign, a successful geo-lift test requires rigorous statistical planning to ensure the results are valid, reliable, and provide a true measure of causal impact.

Running a successful geo-lift test requires rigorous planning and execution. Here is a detailed, step-by-step guide for ecommerce brands, with a focus on the Dutch market.

Step 1: Define Your Hypothesis

Start with a clear question. What do you want to learn? A good hypothesis is specific, measurable, and actionable.

Step 2: Select Your Geographic Markets

This is the most critical step. The validity of your test depends on the comparability of your test and control groups. The goal is to create two groups of geos that have behaved similarly in the past.

• Data Sources: Use historical sales data from your Shopify store, broken down by province or city. You can also enrich this with demographic data from sources like the Dutch Centraal Bureau voor de Statistiek (CBS) [1]. * Matching Techniques: For a simple test, you can use a manual matching process. For more advanced analysis, techniques like matched-market analysis use statistical algorithms to find the best possible pairings. The key is to match on key variables like historical sales volume, population size, and customer demographics. * Example for the Netherlands: You could create a test group of provinces (e.g., Noord-Holland, Zuid-Holland) and a control group of similar provinces (e.g., Utrecht, Noord-Brabant). The key is to ensure both groups have a similar pre-test sales trajectory.

Step 3: Determine Test Duration and Budget

Your test needs to run long enough to produce statistically significant results. This depends on your sales volume and the expected lift.

• Power Analysis: A statistical power analysis is the best way to determine the required sample size and duration. There are online calculators that can help with this. A small, underfunded test is unlikely to produce a detectable lift.

Step 4: Execute the Campaign

Launch your campaign, targeting only the geographic areas in your test group. It is important to ensure there is no "spillover" or contamination into the control group. Double-check your campaign’s location targeting settings.

Step 5: Analyze the Results

Once the test is complete, it is time to measure the lift. The basic formula is:

Incremental Sales = (Test Group Sales During Test) - (Predicted Control Group Sales During Test)

To get the predicted sales, you can use a simple pre-post comparison, but a more reliable method is to use a Difference-in-Differences (DiD) analysis. This technique, detailed in studies from institutions like Stanford [2], compares the change in sales in the test group to the change in sales in the control group, which helps to control for seasonality and other time-based trends.

DiD Lift = (Test_Post - Test_Pre) - (Control_Post - Control_Pre)

From here, you can calculate your true, incremental ROAS:

Incremental ROAS = Incremental Sales / Ad Spend

This is the number that matters. This is the number your CFO trusts.

While the principles of geo-lift testing are useful, manual execution is complex, time-consuming, and prone to error.

For more insights on how to sharpen your marketing mix, check out our post on /blog/marketing-mix-model-shopify or learn about the dangers of blended ROAS in /blog/blended-roas-lie-track-instead.

Frequently Asked Questions (FAQ)

What is the difference between a geo-lift test and an A/B test?

An A/B test compares two versions of a single element, like a headline or an image, at the user level. A geo-lift test compares the impact of an entire campaign across different geographic regions. Geo-lift tests are designed to measure the causal impact of marketing on sales, while A/B tests are typically used for conversion rate refinement.

How long should I run a geo-lift test?

The duration of a geo-lift test depends on your sales cycle and the expected impact of your campaign.

Can I run geo-lift tests for all my marketing channels?

Yes, you can run geo-lift tests for any channel where you can target your advertising geographically. This includes social media, search, display, and even offline channels like radio and television. The key is the ability to isolate a specific geographic area for your campaign, which is a feature of most modern advertising platforms.

What if my business is too small for a geo-lift test?

While larger businesses with more data will see results faster, even smaller brands can benefit from geo-lift testing. The key is to design the test carefully and be realistic about the expected lift. A smaller lift will require a longer test duration to become statistically significant, but the insights gained are just as valuable.

Is geo-lift testing expensive?

The cost of a geo-lift test is simply the cost of the advertising you are running. The real question is: what is the cost of not running a geo-lift test? Continuing to spend money on channels that do not drive incremental sales is far more expensive in the long run. Geo-lift testing is an investment in efficiency.

References

[1] Centraal Bureau voor de Statistiek. (n.d.). StatLine. Retrieved from https://opendata.cbs.nl/statline/#/CBS/nl/

[2] Athey, S., & Imbens, G. W. (2017). The State of Applied Econometrics: Causality and Policy Evaluation. Stanford University.

[3] Abadie, A. (2021). Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects. Journal of Economic Literature, 59(2), 391-425. Retrieved from https://www.aeaweb.org/articles?id=10.1257/jel.20191450