What agents get wrong about attribution data
Four failure modes recur when a model summarises measurement. Each has a specific fix, and three of them are schema problems rather than prompt problems.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 3 min read
The four
| Failure | What it looks like | Fix |
|---|---|---|
| Dropped uncertainty | "Meta returned 2.4" from an estimate spanning 1.1 to 3.7 | Put the interval in the schema and require it in output |
| Ranking the unrankable | An ordered list of channels whose intervals all overlap | Require overlap checking before any ordering |
| Filling the floor | A confident figure for a channel marked unmeasurable | Keep the label in the payload, not just the notes |
| Stale window | Summarising an old file as though it were current | Put the window in the payload as a field |
Dropped uncertainty is the common one
A model asked to write a crisp summary will write a crisp summary, and hedges are the first thing that goes. If the confidence interval is present in the data, an instruction to carry it into every channel mention resolves most of it.
The instruction has to be explicit. "Be accurate" does not produce hedging; "state the range whenever you state an estimate" does.
Ranking the unrankable
Given a list of numbers, a model will order them, because ordering is what one does with numbers. If three channels have overlapping intervals, the ordering is not supported by the data, and a confidently ranked list is a fabrication produced by a reasonable-looking operation.
Requiring the model to check overlap before ordering, and to group indistinguishable channels together, fixes it. This is also a good habit for humans.
Filling the floor
If the unmeasurable label lives in a footnote rather than in the record, the model does not see it and produces a number for a channel that has none. Keep it in the payload as a field on the channel. The reason channels get this label is in the measurability floor.
Stale window
A file whose period is only in its name will be summarised as current indefinitely. Put the window inside the payload as a first-class field and the problem disappears.
Why these matter more than they look
Because the output reads well. A fabricated ranking with confident prose is more persuasive than an honest one with hedges, and the person reading the summary has no way to tell. That is the case for putting the constraints in the data rather than trusting the reader.
The agent-facing surface is on for AI assistants, and the boundary on what agents should decide is in can an AI agent do your attribution.
Related answers
Frequently asked questions
Why do AI summaries drop confidence intervals?
Because a crisp sentence reads better than a qualified one, and hedges are the first thing summarisation pressure removes. An explicit instruction to state the range whenever an estimate is stated resolves most of it.Why do models rank channels that cannot be ranked?
Because ordering is the obvious operation on a list of numbers. If several channels have overlapping intervals, the ordering is not supported by the data, and the confident list is a fabrication produced by a reasonable-looking step.How do I stop a model summarising a stale report?
Put the analysis window inside the payload as a first-class field rather than leaving it in the filename. A file whose period is only in its name will be summarised as current indefinitely.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- ShopifyShopify is an ecommerce platform for creating online stores and selling products. Attribution modeling shows which marketing channels drive traffic and conversions within Shopify.