How to test the gap between reported and causal ROAS: A gap between two numbers is a hypothesis, not a finding. Three ways to settle which one was closer to the truth, ordered by what each one costs you.
Read the full article below for detailed insights and actionable strategies.
The numbers behind the problem
Avg ad spend wasted
Meta ROAS inflation
Cost to find out
Setup time
A gap between reported and causal ROAS is a hypothesis. Until you test it, you have two numbers and a preference. Three designs settle it, and they differ mainly in what they cost you.
The three, by cost
| Design | What it costs | What it settles |
|---|---|---|
| Spend step change | Nothing, if you were adjusting anyway | Whether revenue moves as one number predicts |
| Time-based pause | The channel's revenue for the dark period | The same, more sharply |
| Geo holdout | Revenue forgone in held-out regions | It, properly, with a comparison group |
The free-ish one: a spend step
If you are going to change a channel's budget anyway, size the change deliberately and record what each number predicts beforehand. Reported ROAS implies a certain revenue change; the causal estimate implies a smaller one. Then look.
The prediction has to be written down first. Written afterwards it is a story, and everyone reading it will know.
The pause
Turn one channel off for at least two full purchase cycles, having chosen the read window in advance. Shorter pauses still contain the channel's delayed effect, which produces a comfortable result that means nothing. The design points are in how to measure incremental lift.
The holdout
Split by geography, keep the channel running in some regions and dark in others, and compare. This is the only one of the three with a genuine comparison group, which is why it is the strongest and the most expensive. The geo testing guide covers the pre-period and the sizing.
What a result actually tells you
If revenue falls by roughly what the causal estimate predicted, the causal estimate was closer for that channel in that period. That is a narrower claim than "causal attribution is right", and stating it narrowly is what makes it credible.
Repeat it on a second channel and a second period before generalising. One test on one channel is one observation, and the temptation to draw a company-wide conclusion from it is strong and should be resisted.
The calibration that comes out of it
Over a few tests you will develop a rough factor per channel: reported ROAS on branded search runs some multiple of what a test shows, and so on. That factor is more useful than either raw number, because it lets you read the platform dashboards you cannot stop looking at with the right correction applied.
Where the cheap read fits
Use an observational read to pick which channel to test, because testing everything is unaffordable. Causality Engine produces that read from a Google Analytics export at €99 once, with confidence intervals, coverage and a design label per channel, refundable if it does not move a budget decision.
Then spend the expensive instrument once, on the channel where being wrong costs the most. That order is argued in cut the channel you would holdout first.
The discipline that makes any of it work
One change at a time, the read window chosen in advance, and the prediction written down before the data arrives. Without those three the test produces a number and no evidence.
Related answers
Get attribution insights in your inbox
One email per week. No spam. Unsubscribe anytime.
Key Terms in This Article
Analytics
Analytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Causal Attribution
Causal Attribution uses causal inference to determine which marketing touchpoints genuinely cause conversions, not just correlate with them.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Confidence Interval
Confidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
Dashboards
Dashboards are graphical user interfaces that provide at-a-glance views of key performance indicators (KPIs). They monitor campaign performance and visualize attribution insights.
Google Analytics
Google Analytics is a web analytics service that tracks and reports website traffic.
Incrementality
Incrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.