What Happens When Companies Actually Test Their Ads

Over the past decade, some of the largest advertisers on earth ran controlled experiments on their own spend. The results are public, they are consistent, and they all point the same way: measured incremental lift is far below the lift the platforms attributed.

"So let that sink in.

You are almost certainly not spending half a billion. But the lesson scales down: if the demand is already yours, a large share of the spend chasing it is not incremental.

eBay, 2015 (Blake, Nosko and Tadelis)

In a controlled study of paid brand search, eBay found the incremental effect was near zero. The people clicking branded search ads were going to reach eBay anyway. The ads were paying to be present at a sale they did not cause.

Meta's own research, 2019 (Gordon et al., 15 large RCTs)

Using fifteen large randomized experiments on its own platform, Meta's researchers found that common attribution approaches routinely overstated true lift, often several-fold. The platform's own scientists documented the gap between attributed and causal.

Uber, 2017

Installs did not meaningfully drop. Most of that budget had been claiming installs that were happening regardless.

Airbnb, 2020

The demand was largely already there; the spend had been harvesting it.

P&G and Chase

JPMorgan Chase reduced its display footprint from 400,000 sites to 5,000 and saw the same results. Less spend, spread across far fewer places, produced the same outcome.

What they have in common

Every one of these is the same finding at different scale: attributed lift and measured lift are not the same number, and the gap runs in one direction. The dashboards over-credited. When the spend was removed under controlled conditions, the sky did not fall, which means much of it was never incremental.

Takeaway: Across eBay, Meta, Uber, Airbnb, P&G and Chase, controlled tests show attributed lift far exceeds real lift. Your account is not the exception.

References: Blake, Nosko and Tadelis, Econometrica 2015 · Gordon et al. 2019 (Meta) · Uber 2017 · Airbnb 2020 · P&G and Chase public disclosures.

Watch the full argument above, or on YouTube. Understand the mechanism in The 4 Reasons Your Attribution Numbers Lie, and see when DIY measurement is no longer enough in DIY Attribution vs Causal Measurement.