Ask Your Attribution Vendor for a Placebo Test

Between 14 August and 2 September 2026, The Price of Being Found audited the public materials of 31 commercial measurement vendors for a published comparison of the vendor's estimates against randomised experimental results, with a sample, a design and an interval. It found none. The book is careful about what that means: not that no vendor has ever validated, not that any method is wrong, only that no such publication was in public materials on those dates. It also means the Black Friday read your vendor will hand you in December rests on a validation you have not seen.

The question with no prepared answer is the placebo one: what would this system report if my advertising had no effect at all? Octalysis would file the moment you ask it under Unpredictability, because you genuinely do not know what the vendor will say, and neither, often, do they.

What the audit did and did not find

For each vendor: the website, the documentation, the research section, the case studies. Counted: a comparison against real randomised experiments, sample disclosed, design stated, discrepancies reported. Not counted: backtests, simulated data, accuracy claims without a disclosed comparison, client testimony. The book invites any of the 31 to send a published validation it missed, promising to print it with an apology, and calls that its own falsification condition.

Google's Meridian and Meta's Robyn both designate real randomised experiments as the calibration ground truth; an earlier draft had claimed otherwise and withdrew it.

The sentence from Google's own documentation

The book quotes Meridian's page on assessing a fitted model, retrieved 4 September 2026 and dated 30 June 2026 on the page: directly validating the quality of causal inference is difficult and requires well-designed experiments, and since you are using an MMM, experiments are likely not practical. The consequence, in the same passage, is that the causal inference cannot be directly assessed and you have to rely on indirect measures. The book's gloss: bathroom scales sold with a note that they can only be checked against a second set of scales you have been told you cannot have. Consistent, every morning. Not the same thing as right.

Four questions to ask before the Cyber Week read

The book's list runs to thirteen and is addressed to any vendor, including its own publisher. These four are the ones that matter most before a peak:

1. Have you validated this method against randomised experiments on real data, not simulated, not backtested? Show me the study, the sample, and the discrepancies.
2. What result would this system produce if my advertising had no effect at all? Have you tested that?
3. What is the confidence interval, not the point estimate?
4. Which of my channels are not measurable at my current spend, on your method? The book's note on this one: a vendor who says all of them are measurable has just failed the measurability chapter.

What an honest answer sounds like

To question two: a description of a placebo run, the model applied to a period or a region where nothing was on, with the null result it produced, or an admission that it has not been done and a date by which it will be. To question four: a list, and the book says it is not short for most advertisers. Question four is the one that separates a measurement partner from a measurement salesperson.

The same standard applies to a causal read from this site. Ask it the placebo question too.

What to do this week

• If you have to defend the number: send the four questions to the vendor in writing this week, so the answers exist before the December read does. An answer that arrives with the read is a rationalisation; one that arrives in September is a specification.
• If you own the budget: ask question four alone. The list you get back is the list of channels the December report will describe with false precision.

The calendar has the dates. One number for the Black Friday planning slide is what a read that passes these questions looks like on a page.

As of 9 September 2026. The 31-vendor audit, the Meridian quotation and the vendor questions are from The Price of Being Found (Edition 2.10), Chapter 17, with the book's stated limitations; the audit covered public materials only.