How to test the gap between reported and causal ROAS
A gap between two numbers is a hypothesis, not a finding. Three ways to settle which one was closer to the truth, ordered by what each one costs you.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 3 min read
Run the numbers for your store: the free break-even ROAS calculator.
A gap between reported and causal ROAS is a hypothesis. Until you test it, you have two numbers and a preference. Three designs settle it, and they differ mainly in what they cost you.
The three, by cost
| Design | What it costs | What it settles |
|---|---|---|
| Spend step change | Nothing, if you were adjusting anyway | Whether revenue moves as one number predicts |
| Time-based pause | The channel's revenue for the dark period | The same, more sharply |
| Geo holdout | Revenue forgone in held-out regions | It, properly, with a comparison group |
The free-ish one: a spend step
If you are going to change a channel's budget anyway, size the change deliberately and record what each number predicts beforehand. Reported ROAS implies a certain revenue change; the causal estimate implies a smaller one. Then look.
The prediction has to be written down first. Written afterwards it is a story, and everyone reading it will know.
The pause
Turn one channel off for at least two full purchase cycles, having chosen the read window in advance. Shorter pauses still contain the channel's delayed effect, which produces a comfortable result that means nothing. The design points are in how to measure incremental lift.
The holdout
Split by geography, keep the channel running in some regions and dark in others, and compare. This is the only one of the three with a genuine comparison group, which is why it is the strongest and the most expensive. The geo testing guide covers the pre-period and the sizing.
What a result actually tells you
If revenue falls by roughly what the causal estimate predicted, the causal estimate was closer for that channel in that period. That is a narrower claim than "causal attribution is right", and stating it narrowly is what makes it credible.
Repeat it on a second channel and a second period before generalising. One test on one channel is one observation, and the temptation to draw a company-wide conclusion from it is strong and should be resisted.
The calibration that comes out of it
Over a few tests you will develop a rough factor per channel: reported ROAS on branded search runs some multiple of what a test shows, and so on. That factor is more useful than either raw number, because it lets you read the platform dashboards you cannot stop looking at with the right correction applied.
Where the cheap read fits
Use an observational read to pick which channel to test, because testing everything is unaffordable.
Then spend the expensive instrument once, on the channel where being wrong costs the most. That order is argued in cut the channel you would holdout first.
The discipline that makes any of it work
One change at a time, the read window chosen in advance, and the prediction written down before the data arrives. Without those three the test produces a number and no evidence.
Related answers
Frequently asked questions
How do I know whether reported or causal ROAS is closer to the truth?
Run a test. Predict the revenue change each number implies, write both predictions down beforehand, then change the channel's spend and compare. A geo holdout settles it most cleanly because it has a genuine comparison group.Can one test tell me my attribution method is right?
It tells you which estimate was closer for that channel in that period, which is a narrower claim and the only one the evidence supports. Repeat on another channel and another period before drawing anything general.What is a calibration factor for reported ROAS?
A rough per-channel multiple relating what a platform reports to what a test shows. Once you have it, you can keep reading platform dashboards with the correction applied, which is more practical than trying to stop looking at them.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Causal AttributionCausal Attribution uses causal inference to determine which marketing touchpoints genuinely cause conversions, not just correlate with them.
- DashboardA dashboard is a visual display of key information required to achieve specific objectives. It consolidates data onto a single screen for quick review.
- DashboardsDashboards are graphical user interfaces that provide at-a-glance views of key performance indicators (KPIs). They monitor campaign performance and visualize attribution insights.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- RevenueRevenue is the total income generated by the sale of goods or services related to a company's primary operations.