How to decompose what actually sells: A practical sequence for separating concept from execution across a product line, using batches, a holdout and an interval rather than a hunch about which feature mattered.
Read the full article below for detailed insights and actionable strategies.
Attribution by the numbers
Avg ad spend wasted
Meta ROAS inflation
Cost to find out
Setup time
Separating concept from execution takes a batch design, not a better eye. Here is the sequence.
Step one: write the hypothesis down first
Name the property you believe is doing the work, before you design anything. "The subject matters more than the style" is a hypothesis. "This design is great" is not.
Writing it down first is what stops the result from being reinterpreted afterwards to fit whatever happened.
Step two: build the batch to vary one thing
Take one concept and produce several executions of it. Take one execution style and apply it to several concepts. Two batches, each holding one axis fixed.
The batch has to be large enough that a single item's luck does not decide the result. Small batches produce confident nonsense.
Step three: keep the traffic allocation fixed
This is the step that fails most often. If you push more traffic to the items that start well, you have replaced your experiment with the feedback loop it was designed to avoid.
Fix the allocation in advance and leave it alone for the whole window, even when one item looks like it is winning.
Step four: read the interval
| Result | Reading |
|---|---|
| Concept batch separates, execution batch does not | Concept is doing the work |
| Both separate | Both matter, size them before choosing |
| Neither separates | Underpowered, or the effect is small |
| Confidence interval spans zero | Not resolved, do not conclude |
The last row is the one to respect. An unresolved result is a real outcome and it means run it longer or bigger, not pick the higher midpoint.
What this does not cover
Channel effects. If one batch happened to get more of a particular traffic source, the comparison is contaminated. That is the same incrementality question the channel work answers.
A causal read on a Google Analytics export returns per channel an estimate, an interval and a coverage share for 99 euro once, refunded if it does not move a budget decision, which tells you whether your traffic mix shifted under the test. The interactive demo shows it with no signup.
Related answers
Key Terms in This Article
Analytics
Analytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Confidence Interval
Confidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
Google Analytics
Google Analytics is a web analytics service that tracks and reports website traffic.
Incrementality
Incrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
Traffic Source
Traffic Source is the origin through which users find a site. Common sources include organic search, paid search, direct traffic, and referrals.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.
Frequently Asked Questions
How do I separate concept from execution in a product line?
Build two batches. One holds the concept fixed and varies execution; the other holds execution fixed and varies concept. Each must be large enough that one item's luck does not decide the outcome.
Why fix traffic allocation during the test?
Because pushing traffic toward early winners recreates the feedback loop the test exists to avoid. Fix the allocation in advance and leave it for the whole window, even when one item looks like it is winning.
What if neither batch separates?
The test was underpowered or the effect is small. Run it longer or larger. Picking the higher midpoint from an unresolved result is not a conclusion.