Confirm a weak channel with a holdout test
The read narrows it to one channel; a holdout settles it. How to size and run the test without spending more revenue than the answer is worth.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 3 min read
Run the numbers for your store: the free holdout test planner.
A holdout is the instrument that settles it, and the whole skill is sizing it so the answer costs less than the decision is worth.
The sizing decisions
| Decision | Rule of thumb |
|---|---|
| How much to hold out | Enough regions or share that the difference can be detected |
| For how long | At least two full purchase cycles, plus a pre-period |
| What to hold constant | Everything else, especially promotions |
| The read window | Written down before the test starts |
Geographic beats temporal
If you can split by region, do. A geo split holds time constant, which removes seasonality, promotions and competitor activity from the comparison in one move. A time-based pause has to assume the two periods were otherwise comparable, and for most brands they are not.
The mechanics of the split, including the pre-period you need to establish that the groups behave alike, are in the geo testing guide.
The cost, stated honestly
You forgo whatever the channel would have generated in the held-out regions for the duration. That is a real cost and it is the reason to run one holdout on the channel with the most at stake rather than several on small ones. The argument for that ordering is in cut the channel you would holdout first.
Reading the result
Compare the held-out group against the running group over the read window you fixed in advance. The difference is your lift estimate, and it comes with its own uncertainty, which should be reported alongside it.
If the result is consistent with the observational read you started from, you now have two independent instruments agreeing, which is a much stronger position than either alone. If it disagrees, the holdout wins, because its design is stronger.
What a holdout cannot fix
It cannot tell you whether the channel would work with different creative, at a different budget, or in a different season. It answers a specific question about a specific configuration, and generalising beyond that is where results get misused.
Where the cheap read fits
Before the test, to choose the channel. The comparison of the two instruments is in incremental ROAS without a geo test.
The discipline that decides it
The read window and the decision threshold both written down before the test starts. Chosen afterwards, they are conclusions with a procedure drawn around them, and everyone reading the result will discount it accordingly.
Related answers
Frequently asked questions
Should I use a geographic or a time-based holdout?
Geographic where possible, because it holds time constant and removes seasonality, promotions and competitor activity from the comparison. A time-based pause has to assume two periods were otherwise comparable, which for most brands they are not.How long should a holdout run?
At least two full purchase cycles for your brand, plus a pre-period long enough to establish that the groups behave alike. Shorter tests still contain the channel's delayed effect and produce comfortable results that mean nothing.What can a holdout not tell me?
Whether the channel would work with different creative, at a different budget, or in another season. It answers a specific question about a specific configuration, and generalising beyond that is where results get misused.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Ad SpendAd Spend is the total amount invested in advertising campaigns. It is measured against Return on Ad Spend (ROAS) to evaluate campaign effectiveness.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.