Testing if AI referrals are incremental: Three designs that separate assistant referrals creating demand from those merely harvesting demand you already had, ordered by cost and by strength of evidence.
Read the full article below for detailed insights and actionable strategies.
The attribution problem
One sale. Four channels. 400% credit claimed.
Reported revenue: €400 · Actual revenue: €100 · Gap: €300
A referral count cannot tell you whether the order would have happened anyway. Three designs can, at different prices.
Why the count is not the answer
Someone asks an assistant for a recommendation, sees your brand, and buys. Someone else already knew your brand, asked the assistant to confirm, and bought. Both produce one assistant referral and one order. Only the first is incremental.
Budget logic diverges completely between those two cases, and no amount of session detail separates them.
Design one: the branded search control
Watch branded search volume and assistant referrals together over the same window. If assistant referrals rise while branded search holds, that is weak evidence of new demand. If branded search falls as assistant referrals rise, you are probably watching the same demand change its route.
Cheap, fast, and genuinely weak evidence. Useful for ruling things out, not for a budget move.
Design two: the content holdout
Pick a set of answer pages and stop maintaining them, or deliberately keep a comparable set un-updated. Compare assistant-referred sessions and orders across the two sets over a full window.
This is a real test with a real cost: the pages you hold back may lose visibility you wanted. Run it on a set you can afford to lose.
Design three: the geo holdout
The strongest design available and the most expensive. A geo holdout on the activity feeding assistant visibility, held for weeks, settles the question in a way the other two cannot.
Reserve it for the case where being wrong costs more than the test does. The ordering argument is in cut the channel you would holdout first.
Reading whichever you run
Read the interval, not the midpoint. An estimate whose confidence interval spans break-even is a result that says "not resolved," and treating its midpoint as an answer is how confident wrong decisions get made.
A read on a Google Analytics export gives you that interval per channel for 99 euro once, refunded if it does not move a budget decision. The interactive demo shows the shape of the output with no signup.
Related answers
Key Terms in This Article
Analytics
Analytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Confidence Interval
Confidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
Google Analytics
Google Analytics is a web analytics service that tracks and reports website traffic.
Incrementality
Incrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
Session
A Session is a group of user interactions with your website within a given timeframe. It can include multiple page views, events, and transactions.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.
Frequently Asked Questions
How do I test whether AI referrals are incremental?
Three designs, increasing in strength and cost: watch branded search as a control, hold back a comparable set of answer pages, or run a geo holdout on the activity feeding assistant visibility. Only the last settles the question.
Why is a referral count not enough?
A visitor who discovered you through an assistant and one who already knew you and asked for confirmation both produce one referral and one order. The count cannot separate them, and they imply opposite budget decisions.
What does it mean if the confidence interval spans break-even?
That the test did not resolve. The honest reading is not resolved rather than the midpoint, and acting on the midpoint is how a confident wrong decision gets made.