How to decompose what actually sells
A practical sequence for separating concept from execution across a product line, using batches, a holdout and an interval rather than a hunch about which feature mattered.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 2 min read
Run the numbers for your store: the free holdout test planner.
Separating concept from execution takes a batch design, not a better eye. Here is the sequence.
Step one: write the hypothesis down first
Name the property you believe is doing the work, before you design anything. "The subject matters more than the style" is a hypothesis. "This design is great" is not.
Writing it down first is what stops the result from being reinterpreted afterwards to fit whatever happened.
Step two: build the batch to vary one thing
Take one concept and produce several executions of it. Take one execution style and apply it to several concepts. Two batches, each holding one axis fixed.
The batch has to be large enough that a single item's luck does not decide the result. Small batches produce confident nonsense.
Step three: keep the traffic allocation fixed
This is the step that fails most often. If you push more traffic to the items that start well, you have replaced your experiment with the feedback loop it was designed to avoid.
Fix the allocation in advance and leave it alone for the whole window, even when one item looks like it is winning.
Step four: read the interval
| Result | Reading |
|---|---|
| Concept batch separates, execution batch does not | Concept is doing the work |
| Both separate | Both matter, size them before choosing |
| Neither separates | Underpowered, or the effect is small |
| Confidence interval spans zero | Not resolved, do not conclude |
The last row is the one to respect. An unresolved result is a real outcome and it means run it longer or bigger, not pick the higher midpoint.
What this does not cover
Channel effects. If one batch happened to get more of a particular traffic source, the comparison is contaminated. That is the same incrementality question the channel work answers.
Related answers
Frequently asked questions
How do I separate concept from execution in a product line?
Build two batches. One holds the concept fixed and varies execution; the other holds execution fixed and varies concept. Each must be large enough that one item's luck does not decide the outcome.Why fix traffic allocation during the test?
Because pushing traffic toward early winners recreates the feedback loop the test exists to avoid. Fix the allocation in advance and leave it for the whole window, even when one item looks like it is winning.What if neither batch separates?
The test was underpowered or the effect is small. Run it longer or larger. Picking the higher midpoint from an unresolved result is not a conclusion.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Traffic SourceTraffic Source is the origin through which users find a site. Common sources include organic search, paid search, direct traffic, and referrals.