What Happens When Companies Actually Test Their Ads: eBay, Meta, Uber, Airbnb, P&G, Chase. When large advertisers ran controlled experiments, measured lift came in far below attributed lift. The evidence, with sources.
Read the full article below for detailed insights and actionable strategies.
Attribution by the numbers
iOS tracking loss
Google Brand cannibalization
Klaviyo overstatement
TikTok attribution lag
Do not take our word for it, and do not take your dashboard's word for it either. Over the past decade, some of the largest advertisers on earth ran controlled experiments on their own spend. The results are public, they are consistent, and they all point the same way: measured incremental lift is far below the lift the platforms attributed.
Joe walks through the Airbnb case live in the lesson, and it is the one to sit with:
"They cut half a billion, more than half a billion of their budget that was allocated to these channels... and it retained 95% of the traffic."
Joe, Causality Engine Academy (Lesson 1)
"So let that sink in. Half a billion was being spent for a difference of 5%."
Joe, Causality Engine Academy (Lesson 1)
You are almost certainly not spending half a billion. But the lesson scales down: if the demand is already yours, a large share of the spend chasing it is not incremental.
eBay, 2015 (Blake, Nosko and Tadelis)
In a controlled study of paid brand search, eBay found the incremental effect was near zero. The people clicking branded search ads were going to reach eBay anyway. The ads were paying to be present at a sale they did not cause.
Meta's own research, 2019 (Gordon et al., 15 large RCTs)
Using fifteen large randomized experiments on its own platform, Meta's researchers found that common attribution approaches routinely overstated true lift, often several-fold. The platform's own scientists documented the gap between attributed and causal.
Uber, 2017
Uber paused roughly $100M of $150M in app-install spend. Installs did not meaningfully drop. Most of that budget had been claiming installs that were happening regardless.
Airbnb, 2020
Airbnb cut around $540M of performance marketing and retained roughly 95 percent of its traffic. The demand was largely already there; the spend had been harvesting it.
P&G and Chase
P&G cut $200M in digital advertising with no change in sales growth. JPMorgan Chase reduced its display footprint from 400,000 sites to 5,000 and saw the same results. Less spend, spread across far fewer places, produced the same outcome.
What they have in common
Every one of these is the same finding at different scale: attributed lift and measured lift are not the same number, and the gap runs in one direction. The dashboards over-credited. When the spend was removed under controlled conditions, the sky did not fall, which means much of it was never incremental.
Takeaway: Across eBay, Meta, Uber, Airbnb, P&G and Chase, controlled tests show attributed lift far exceeds real lift. Your account is not the exception.
References: Blake, Nosko and Tadelis, Econometrica 2015 · Gordon et al. 2019 (Meta) · Uber 2017 · Airbnb 2020 · P&G and Chase public disclosures.
Watch the full argument above, or on YouTube. Understand the mechanism in The 4 Reasons Your Attribution Numbers Lie, and see when DIY measurement is no longer enough in DIY Attribution vs Causal Measurement.
Get attribution insights in your inbox
One email per week. No spam. Unsubscribe anytime.
Key Terms in This Article
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Click
Click is the action a user takes to interact with a digital advertisement, redirecting them to a website or landing page. Clicks are a fundamental metric for measuring ad engagement and a primary input for click-based attribution models.
Dashboard
A dashboard is a visual display of key information required to achieve specific objectives. It consolidates data onto a single screen for quick review.
Dashboards
Dashboards are graphical user interfaces that provide at-a-glance views of key performance indicators (KPIs). They monitor campaign performance and visualize attribution insights.
Experiments
Experiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
Performance Marketing
Performance Marketing is a digital marketing type where advertisers pay only for specific actions like clicks, leads, or sales.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.
Frequently Asked Questions
What do controlled ad experiments actually show?
Across eBay (2015), Meta's own 15 RCTs (2019), Uber (2017), Airbnb (2020), P&G and Chase, measured incremental lift came in far below attributed lift. Removing spend under controlled conditions did not drop outcomes proportionally.
Did companies lose sales when they cut ad spend?
Largely no. Uber paused about $100M of $150M in app-install spend without a meaningful install drop; Airbnb cut roughly $540M of performance marketing and kept about 95 percent of traffic. Much of the spend was never incremental.