Why Vendor Lift Numbers Are Not Comparable

Lift is a ratio: the treated group against a baseline of what would have happened otherwise. The treated group is observed. The baseline is a choice. Change the comparison group, the window or the design and the same campaign yields a different lift figure without anybody behaving dishonestly. That is why two vendor numbers for one campaign cannot be compared, and why a lift percentage on its own is not information.

The four choices hidden inside any lift figure

The comparison group. Randomised, meaning assignment was by chance, or constructed, meaning someone selected or modelled a group intended to resemble the treated one. Randomised is far stronger. Constructed baselines inherit whatever made the two groups different in the first place, and in advertising the treated group is usually the one the targeting judged more likely to buy.

The window. How long after exposure a conversion still counts. A longer window catches more conversions and inflates lift; a shorter one deflates it. Fixed in advance or chosen after seeing the data is the difference between a measurement and a story.

The unit. Lift per exposed user, per impression, per euro of spend, or on total revenue. These produce different numbers from identical data and are routinely quoted interchangeably.

The population. Everyone, or a segment. A lift computed on the most responsive segment is a real number about that segment and is not the campaign's effect.

Why the bigger number is often the weaker one

A lift figure grows when the baseline is lower, and the baseline is lowest when the comparison group is least like the treated group. So a vendor with a loose comparison group will honestly report a larger lift than a vendor with a tight one, measuring the same campaign.

This inverts the intuition that a higher number means a better result. Between two lift figures, the smaller one with a randomised baseline is worth more than the larger one with a modelled baseline, and a buyer comparing headline percentages will pick exactly wrong.

The four properties to demand

Ask for these before comparing anything.

1. Design. Randomised, quasi-experimental or observational. If observational, the figure is a description and should be labelled as one.
2. Interval. The range, not the point. A lift of 12% with an interval from minus 4 to 28 is not a 12% lift; it is a result consistent with no effect.
3. Minimum detectable effect. The smallest effect the design could have found at that volume. If the reported lift is near the floor, the study could not have distinguished it from noise.
4. Coverage. What share of real orders the underlying data could see.

These are the four properties of a defensible number, and a vendor who cannot supply them for their own study is not being evasive so much as revealing that the study was not built to be reviewed. How to vet an attribution vendor before you sign turns this into a checklist, and attribution vendors that explain their methodology openly covers what open actually means.

The audit finding worth knowing

The Price of Being Found audited the public materials of 31 commercial measurement vendors between 14 August and 2 September 2026 and found none with a published validation against randomised experiments, with a disclosed sample, design and discrepancies. That is a finding about public materials on those dates rather than a claim that no such work exists privately. It does mean that when you ask for the design behind a lift number, you are usually asking for something that has never been written down.

What to do this week

• If you have to defend the number: refuse to put a vendor lift figure in a deck without its design and interval beside it. If those are unavailable, the figure is not reportable.
• If you own the budget: ask the placebo question. What would your system report if this campaign had no effect at all, and have you tested that? Ask your vendor for a placebo test covers the exchange.

The same standard applies to this site's publisher. The interactive demo shows a read that carries an interval and a stated design on every estimate.

The 31-vendor audit and the four properties are from The Price of Being Found (Edition 2.10), Chapters 15 and 17, with the book's stated limitations: the audit covered public materials only.