Testing if AI referrals are incremental
Three designs that separate assistant referrals creating demand from those merely harvesting demand you already had, ordered by cost and by strength of evidence.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 2 min read
A referral count cannot tell you whether the order would have happened anyway. Three designs can, at different prices.
Why the count is not the answer
Someone asks an assistant for a recommendation, sees your brand, and buys. Someone else already knew your brand, asked the assistant to confirm, and bought. Both produce one assistant referral and one order. Only the first is incremental.
Budget logic diverges completely between those two cases, and no amount of session detail separates them.
Design one: the branded search control
Watch branded search volume and assistant referrals together over the same window. If assistant referrals rise while branded search holds, that is weak evidence of new demand. If branded search falls as assistant referrals rise, you are probably watching the same demand change its route.
Cheap, fast, and genuinely weak evidence. Useful for ruling things out, not for a budget move.
Design two: the content holdout
Pick a set of answer pages and stop maintaining them, or deliberately keep a comparable set un-updated. Compare assistant-referred sessions and orders across the two sets over a full window.
This is a real test with a real cost: the pages you hold back may lose visibility you wanted. Run it on a set you can afford to lose.
Design three: the geo holdout
The strongest design available and the most expensive. A geo holdout on the activity feeding assistant visibility, held for weeks, settles the question in a way the other two cannot.
Reserve it for the case where being wrong costs more than the test does. The ordering argument is in cut the channel you would holdout first.
Reading whichever you run
Read the interval, not the midpoint. An estimate whose confidence interval spans break-even is a result that says "not resolved," and treating its midpoint as an answer is how confident wrong decisions get made.
A read on a Google Analytics export gives you a data-health score per channel in place of that interval, for 99 euro once, refundable within 30 days, no questions asked.
Related answers
Frequently asked questions
How do I test whether AI referrals are incremental?
Three designs, increasing in strength and cost: watch branded search as a control, hold back a comparable set of answer pages, or run a geo holdout on the activity feeding assistant visibility. Only the last settles the question.Why is a referral count not enough?
A visitor who discovered you through an assistant and one who already knew you and asked for confirmation both produce one referral and one order. The count cannot separate them, and they imply opposite budget decisions.What does it mean if the confidence interval spans break-even?
That the test did not resolve. The honest reading is not resolved rather than the midpoint, and acting on the midpoint is how a confident wrong decision gets made.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AnalyticsAnalytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- Google AnalyticsGoogle Analytics is a web analytics service that tracks and reports website traffic.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- SessionA Session is a group of user interactions with your website within a given timeframe. It can include multiple page views, events, and transactions.