A Shopify lift test you can run this week
The cheapest genuine experiment available to a Shopify brand: pause one channel, for long enough, with the read window decided before you start. How to size it.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 4 min read
Run the numbers for your store: the free holdout test planner.
The cheapest genuine experiment available to a Shopify brand is a pause: turn one channel off, for long enough, having written down the read window before you start. It is weaker than a geo holdout and far stronger than reading a dashboard, and it costs one channel's spend for a fortnight.
The reason it works is that the comparison is constructed deliberately rather than found. The reason it often fails is that people run it for a week, look at revenue, and conclude something the design could never have supported.
Sizing it before you start
| Decision | Rule of thumb | Why |
|---|---|---|
| Which channel | The one you most suspect, not the smallest | The smallest cannot resolve |
| How long dark | At least two full purchase cycles | Delayed conversions contaminate short pauses |
| Pre-period | At least as long as the dark period | You need a baseline to compare against |
| Read window | Decided in advance, in writing | Otherwise you will pick the flattering one |
The last row is the one that turns an experiment into a story. If the read window is chosen after seeing the data, the result is whatever window you liked, and everyone in the room knows it.
The delayed conversion problem
Pausing a channel does not stop its effect immediately. People who saw the ad last week still buy this week. A one-week pause therefore measures a week that still contains most of the channel's influence, and produces a reassuring result that means nothing.
Two full purchase cycles is the minimum, and for considered purchases it is longer. Your own repeat interval, visible in Shopify, is the guide. The general design points are in how to measure incremental lift.
What to hold constant
Everything else. Do not run the pause during a promotion, do not change creative on the other channels, and do not launch a new product mid-test. Each of those makes the before and after incomparable, and there is no statistical repair.
If holding everything constant is impossible, which it often is in a small brand, that is a reason to prefer a geo split over a time split. Geography lets you hold time constant instead, which is usually easier. The geo testing guide has the mechanics.
Reading the result
Compare orders in the dark period against the pre-period, adjusted for whatever seasonality you can establish. Then be honest about the width of what you have learned. A single pause on a single channel in a single period is one observation, and one observation with an uncontrolled seasonality confound is not a strong result even when the direction is clear.
Pairing it with an observational read on the same window helps, because two weak instruments agreeing is more informative than either alone.
When not to run it
Do not pause a channel that is currently your largest source of new customers during a period you cannot afford to lose. The test is cheap in tooling and not free in revenue, and running it on the wrong channel at the wrong time is a real cost. Use an observational read first to decide which channel is worth the fortnight, which is the argument in cut the channel you would holdout first.
Related answers
Frequently asked questions
How long should I pause a channel to measure its lift?
At least two full purchase cycles for your brand. Shorter pauses still contain most of the channel's delayed influence, so they produce a reassuring result that the design could never actually support. Your Shopify repeat interval is the guide.Should I decide the read window before or after the test?
Before, in writing. A window chosen after seeing the data is a window chosen for its result, and everyone reading the conclusion will discount it accordingly, usually correctly.Is a geo split better than a pause test?
Usually, because it holds time constant rather than requiring you to hold everything else constant across two periods. Geo splits are harder to set up and much easier to interpret, particularly for a small brand where promotions and launches are frequent.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- DashboardA dashboard is a visual display of key information required to achieve specific objectives. It consolidates data onto a single screen for quick review.
- Multi-Touch AttributionMulti-Touch Attribution assigns credit to multiple marketing touchpoints across the customer journey. It provides a complete view of channel impact on conversions.
- RevenueRevenue is the total income generated by the sale of goods or services related to a company's primary operations.
- ShopifyShopify is an ecommerce platform for creating online stores and selling products. Attribution modeling shows which marketing channels drive traffic and conversions within Shopify.