Designing a holdout an agent will respect
How to choose a holdout set for agent-driven catalogue changes so that it stays comparable, stays excluded, and survives contact with an automated process.
By Joris van Huët, Founder & CEOUpdated 2 min read
Run the numbers for your store: the free holdout test planner.
A holdout only works if it is chosen before results exist and excluded by something the agent cannot quietly override. Both halves fail regularly.
Choosing the set
Comparable, not random-looking. The held-back products should resemble the changed ones on the things that drive sales: price band, category, age in the catalogue, recent velocity.
A holdout made of your slowest products is not a control. It is a different population, and the comparison will show a difference that was there before you started.
Sizing it
Big enough that its own noise does not swamp the effect you are looking for. Small enough that the forgone upside is affordable.
There is no universal fraction. The honest version is that a holdout too small to resolve anything is a cost with no information, and a read that reports a confidence interval spanning break-even is usually telling you the design was underpowered rather than that the change did nothing.
Making the exclusion stick
| Weak exclusion | What goes wrong |
|---|---|
| A note in a doc | The agent never reads it |
| A tag the agent can edit | It gets rewritten in a later batch |
| Manual re-checking | Works twice, then nobody does it |
| A hard filter in the agent's instructions plus a post-batch diff | Holds |
The last row is the one that survives. Instruct the exclusion, then verify it after each batch by diffing the held-back set against its pre-batch state. An agent that was told to skip 50 products and skipped 47 has quietly ended your experiment, and only the diff will tell you.
Ending it properly
Decide the end date before you start, and read at the end date rather than when the numbers look interesting. Stopping a test early because it currently looks good is how a random fluctuation becomes a company policy.
The read
The stronger design, when the decision is expensive enough, is a geo holdout.
Related answers
Frequently asked questions
How do I choose a holdout set for catalogue changes?
Pick products comparable to the changed ones on price band, category, catalogue age and recent velocity. A holdout made of your slowest products is a different population, not a control.How do I stop an agent from changing the holdout?
Instruct the exclusion and then verify it with a post-batch diff against the pre-batch state. An agent told to skip 50 products that skipped 47 has ended the experiment, and only the diff reveals it.When should I read the result?
At the end date you set before starting. Stopping early because the numbers currently look good is how a random fluctuation turns into company policy.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.