What is a holdout group in marketing?
A holdout group is a randomly chosen part of your audience that is deliberately kept away from one campaign, channel or email. Its purchases show what would have happened anyway, so the gap to everyone else is what that marketing caused.
By Joris van Huët, Founder & CEOUpdated 7 min read
Run the numbers for your store: the free holdout test planner.
A holdout group is a randomly chosen slice of your audience that you deliberately keep away from one campaign, channel or email. Everyone else gets it as usual. If the people who got it buy more than the holdout, the gap is what that marketing caused. That gap, not the dashboard's count, is the number to budget on.
Meta's help puts it plainly. People who match your ad audience but are intentionally withheld from seeing your ads form your holdout group, also called the control group. Google prefers the second name. In a Google Conversion Lift study, people who see your ads are the treatment group and people who don't are the control group.
A holdout is not an A/B test. An A/B test pits two versions against each other, so everyone still gets something. Meta draws the same line: A/B tests do not include randomized holdout groups. A holdout asks the blunter question. What happens with nothing at all?
The group can live in three places. Inside an ad platform, Meta or Google splits the audience for you. On a map, ads go dark in some regions. On your customer list, you split it yourself and keep one slice out of the campaign.
What one store's data shows
One store's anonymised GA4 export, 1 January 2024 to 21 August 2026. It holds shares of revenue only: no ad spend, no order counts.
| What the export shows | Share of revenue | Source cell |
|---|---|---|
| Journeys with 1 touch: 14 distinct paths, 0.5 days to buy | 79.5% | Journeys sheet, 1 touch row |
| Journeys with 10 or more touches: 1,792 distinct paths | 3.0% | Journeys sheet, 10+ touches row |
| Journeys with 2 or more touches: 3,656 of 3,670 distinct paths | 20.6% | Journeys sheet, 2-3, 4-9 and 10+ touches rows added |
The usual answer stops at the definition. One store's export shows why the definition matters: a path records who touched a sale, not who caused it.
Start with the variety. The Journeys sheet lists 3,670 distinct path sequences, and 3,656 of them have two or more touches. Together those multi-touch paths hold 20.6% of revenue on the Journeys sheet, about a fifth. The 14 one-touch paths on the Journeys sheet hold 79.5%.
So the busy part of the map is not where the money is. The 10+ row on the Journeys sheet makes the point sharpest: 1,792 distinct paths and 3.0% of revenue. A channel can turn up in hundreds of long paths and add nothing. Another can sit alone on a short path, right before a sale that was coming anyway.
The quick paths raise the same doubt from the other side. One-touch journeys on the Journeys sheet took 0.5 days to buy. Last click hands that single touch the whole sale. Nothing in the export says the buyer needed it.
That is the gap a holdout group closes. Remove the touch for a random slice, keep it for everyone else, and count every purchase in both groups. Meta's lift tests count all conversions in the test and holdout groups, clicked or not. The export, by contrast, is a ledger of touches. It cannot run that comparison for you.
Why isn't "people who didn't see the ad" a holdout group?
Because nobody chose them at random. The platform chose who saw your ads, and it aimed at people likely to buy. Compare viewers with non-viewers and you measure that aim along with the ad.
Researchers have checked how far off that gets. A study of 15 advertising experiments at Facebook set the randomised results against the observational methods most advertisers use instead. The observational methods often failed to match the experiments, even with extensive demographic and behavioural data.
Three rules keep a holdout honest:
- Chance picks, before the start. Nobody hand-picks the holdout, and nobody switches sides halfway. Meta keeps people in their test or holdout group for the entire test duration.
- Count everyone you assigned. Meta's test group includes people who were eligible for an ad but never actually saw one. Drop them and the targeting bias walks straight back in.
- Score on all purchases. Use your store's own orders, not the platform's attributed conversions. A holdout that only counts clicked sales measures the click, not the ad.
What can a holdout group not tell you?
- What a bigger budget would do. A holdout prices the campaign at today's spend. Doubling it is a new question with a new answer.
- Why it worked. The gap tells you whether, not which creative, audience or offer did the work.
- Anything for free. Google says every incrementality study comes with an opportunity cost: the held-back people miss ads that might have sold to them. Smaller holdouts cost less but need a longer study. Size is its own question, covered in how big your holdout should be.
- A clean answer through a leak. If held-back people still meet the same ads through another campaign, the gap shrinks. Google warns that holdback users might still see your unmeasured ads.
What to do this week
- Name the touch you trust least. In GA4, click Advertising, then Attribution paths under Attribution, as Google's attribution guide describes. Look at the late touchpoints and find the paid channel that closes the most paths. Pass: you can name one campaign behind it. Fail: you can't, so start from your biggest campaign by spend.
- Check that it can exclude people. In a Meta Sales ad set, go to Audience and select Show more controls. The custom audience exclusion sits there. In Google Ads, open the campaign's Settings and expand Locations. Pass: one of the two routes works. Fail: neither does, so pick a campaign that allows one.
- Write the rule before you test. One line: what you hold back, from whom, for how long, and which number decides. Pass: it is written and dated before launch. Fail: you plan to decide after seeing the result.
Check the homework. Your GA4 Attribution paths export already holds the evidence. Causality Engine reads that one file and shows what each channel caused next to what last-click gave it, in 1 to 2 minutes, for €99 once (excluding VAT), refundable within 30 days. Check the homework
Sources, 1 October 2026: About lift and holdouts in Facebook advertising tests (Meta Business Help Center); About Conversion Lift (Google Ads Help); Differences between Conversion Lift test results and other reporting tools (Meta Business Help Center); A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook (Gordon, Zettelmeyer, Bhargava and Chapsky, Marketing Science, 2019, via IDEAS/RePEc); About Conversion Lift (Meta Business Help Center); Set up Conversion Lift based on users (Google Ads Help); Get started with attribution (Analytics Help); Key events attribution paths report (Analytics Help); How to use custom or lookalike audiences (Meta Business Help Center); Exclude ads from geographic locations (Google Ads Help).
Related answers
Frequently asked questions
Is a holdout group the same as a control group?
Mostly, yes. Meta uses both names for the people withheld from your ads, and Google calls them the control group. The difference shows up in A/B tests, where the control usually gets the old version of something. A holdout gets none of the thing being tested.Do people in a holdout group still see my other marketing?
Yes, and they should. A holdout keeps people away from one thing only, such as one campaign or one email flow. Everything else carries on for both groups, so the gap measures that one thing on top of the rest.Can I use people who never clicked my ad as a holdout group?
No. Non-clickers differ from clickers before any ad runs, and the platform showed your ads to the people it expected to buy. Comparing the two measures that targeting as much as the ad. A holdout must be drawn at random before the campaign starts.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Control GroupControl Group is a segment of an audience intentionally not exposed to a marketing campaign, used to measure the campaign's true causal impact.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
- Google AdsGoogle Ads is an online advertising platform where advertisers bid to display ads, service offerings, and product listings.
- Holdout TestA holdout test is an experiment where a portion of the audience does not see a campaign. This measures the campaign's true incremental impact.
- IncrementalityIncrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
- Treatment GroupTreatment Group is the set of users exposed to a specific marketing intervention. Comparing this group to a control group shows the intervention's causal impact.