Use the methodology doc to design your own check
A published method makes predictions you can test. Three checks built from those predictions, each small enough to run inside an evaluation window.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 3 min read
A published method makes specific predictions about its own behaviour, which means you can test it rather than trust it. Three checks, each of which fits inside an evaluation window.
The three
| Check | What the method predicts | What you do |
|---|---|---|
| Flat window | Wide intervals, because there is nothing to compare | Run it on a period with no spend variation |
| Small channel | No estimate, because it is below the floor | Point it at a channel spending very little |
| Known period | An estimate consistent with what you already know | Run it on a window containing a deliberate change |
The flat-window check
Most methods derive uncertainty partly from how much the input varied. If spend was flat across the window, a well-behaved method should return wide intervals, because there is nothing for it to separate.
A tool that returns tight, confident estimates on a flat window is not deriving uncertainty from the data. That is the single most diagnostic five minutes in an evaluation.
The small-channel check
Point it at a channel spending a small fraction of your budget. A well-built method declines to score it, because its effect is smaller than the ordinary variation in your orders. A tool that produces a confident figure is filling in.
This is not a trick question and vendors should expect it. The arithmetic behind the floor is in the measurability floor.
The known-period check
The strongest of the three. Pick a window containing something you deliberately did: a channel paused, a market where only one channel ran, a large spend step. You know roughly what happened. See whether the output is consistent with it.
Where it disagrees, ask why, and evaluate the explanation rather than the disagreement. Sometimes the tool is right and your memory is wrong, which is itself worth learning. The related design is in the vendor placebo test.
What a good vendor does when you run these
Expects them and helps. A vendor that treats a verification request as an objection is telling you how the rest of the relationship will go.
Running them here
The €99 one-time read on a Google Analytics export is enough for all three, refundable within 30 days, no questions asked, with no subscription running underneath while you evaluate.
The general principle
A method that publishes its behaviour is a method you can hold to it. That is the practical value of documentation, and it is why the absence of documentation is not a neutral fact about a product.
Related answers
Frequently asked questions
How can I test whether an attribution tool is honest?
Run it on a window with flat spend and check whether intervals widen, point it at a channel spending very little and check whether it declines to score, and run it on a period where you already know what happened.Why is a flat window the best quick test?
Because a well-behaved method derives uncertainty partly from how much the input varied. Tight confident estimates on a flat window mean the tool is not deriving uncertainty from your data at all.How should a vendor react to verification requests?
By expecting them and helping. A vendor that treats a request to check its method as an objection is telling you what the rest of the relationship will be like, which is useful information cheaply obtained.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AnalyticsAnalytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Google AnalyticsGoogle Analytics is a web analytics service that tracks and reports website traffic.
- Holdout TestA holdout test is an experiment that keeps a random part of an audience from seeing a campaign and compares its results with those of the part that could see it. The held-out part is the control group, so the gap in outcomes estimates what the campaign added.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Multi-Touch AttributionMulti-Touch Attribution assigns credit to multiple marketing touchpoints across the customer journey. It provides a complete view of channel impact on conversions.
- ReturnsEcommerce returns, or reverse logistics, is the process of customers sending back purchased items to the retailer. Causal analysis helps improve products and reduce return rates.