Incrementality vs Attribution: Two Different Questions

Attribution asks how to divide credit for the sales that happened. Incrementality asks which of those sales would not have happened. These are different questions about different data, and neither one answers the other. The confusion between them is not academic: it is the mechanism by which budget gets moved on evidence that was never about cause.

The clean distinction

Attribution operates on completed journeys. It takes the conversions your system observed, looks at the touchpoints on each path, and applies a rule to share the credit. Every input is something that happened.

Incrementality operates on a comparison. It asks what revenue would have been under a world where the channel was absent, and estimates the difference. That alternative world is not in your data by definition, which is why incrementality needs a design, not just a report.

One divides a known quantity. The other estimates an unknown one.

Why attribution cannot get there from here

The journeys where a channel was absent are missing from the data being divided. A retargeting ad appears in the journeys of people who saw retargeting; the people who would have bought anyway without seeing it are not separated out, because nothing in the observed data separates them.

This is why increasing model sophistication does not close the gap. A data-driven model fits weights on observed journeys more cleverly than a linear rule does, and it is still fitting on the same set that excludes the counterfactual. Causal, rule-based and data-driven attribution in GA4 walks through what each family can and cannot do.

The evidence that the gap is real and large

At eBay, brand-keyword search ads scored as a top channel under attribution logic. Switched off, 99.5% of the forgone paid clicks returned through natural search. Attribution had been describing which door the purchase walked through.

In the Facebook studies The Price of Being Found cites, observational methods overstated effects by roughly three times against randomised ground truth, and six of fourteen comparisons could not be distinguished from zero. The gap is largest where targeting is best, because a system good at finding buyers reports impressive numbers while adding little.

Where each one belongs

Use attribution for operational pacing inside a channel, comparing this week's campaigns against last week's under one consistent rule. The rule cancels out of the comparison, so the direction is readable.

Use incrementality for any decision that moves money between channels, any decision to cut, and any claim made to finance. Incremental versus attributed revenue covers the reporting difference.

The error to avoid is using attributed credit for the reallocation decision, because that is a decision about cause made on a bookkeeping rule.

How to get an incrementality answer

For the channel carrying the most budget, run a randomised holdout: withhold from a randomly chosen slice, fix the window in advance, compare revenue per customer. Check first that the design could detect an effect worth acting on at your volume; below that floor a null result means the test was too small, not that the channel does nothing. Incrementality testing for ecommerce is the playbook.

What to do this week

• If you own the budget: separate your reporting into two columns, attributed and incremental, and leave the second blank where you have no design behind it. The blanks are the honest part.
• If you have to defend the number: when someone asks which channel to cut, ask which column the answer is coming from.

The eBay result and the Facebook comparisons are from The Price of Being Found (Edition 2.10), Chapter 12, with the book's stated caveats.