Retargeting Holdout Before Black Friday: The Mystery Box

Nobody in your company knows what retargeting adds to Black Friday revenue, and the report cannot tell you, because retargeting by construction reaches the people who had already found you. The only way to open the box is a holdout: withhold it from a slice of regions for a fixed window before the peak, compare, and read the interval. There are three possible answers and every one of them changes the plan.

Octalysis calls the pull of a sealed, uncertain outcome the Mystery Box. Most marketing curiosity is idle. This one has a budget line attached and a deadline: for the standard eight-week design, the last start date is 2 October.

Why retargeting is the channel most likely to be flattering itself

The audience is defined by intent it did not create. Someone visited, added to cart, left, and is now being shown the product they already chose. When they buy, the ad was the last thing they saw, and last is where credit lands. The Price of Being Found quotes the experiment that shows the shape of this at its purest: at eBay, switching off brand-keyword search ads lost no measurable short-term sales because 99.5% of the forgone paid clicks came back through natural search. That was brand search, not retargeting, and the book scopes it as such. The structure is the same: an ad shown to people already on their way.

The book's second reference explains why no model fixes it. In the Facebook experiments Gordon and colleagues analysed, the naive comparison of exposed to unexposed users fit the data beautifully and was wrong by a factor of more than four, because the people chosen for exposure were the people most likely to convert. Retargeting is that selection made explicit.

The design that opens the box

A geographic holdout. Pick the unit of geography with enough regions to make inference meaningful. In the Netherlands that is the 40 COROP regions, not the 12 provinces; a province-level test cannot reach p below 0.05 by arithmetic. Hold retargeting out of a subset, keep everything else running, fix the pre-period before you look, and write down the minimum detectable effect and the decision rule before launch.

Then qualify it. Spend share times honest incremental return has to clear the smallest lift your revenue noise allows the design to see, about 8.3% for a typical DTC brand on 26 weeks of history and 8 weeks live. Retargeting is usually a small share of spend, which is exactly why this arithmetic matters: at 2% of revenue and an honest return of 2, the expected effect is 4%, below the floor, and the book's advice is not to run it and to say so. Can a brand your size measure Black Friday lift at all? has the full table. If it does not qualify alone, the honest variants are concentrating the spend so the intensity rises, or testing it bundled with another lower-funnel channel.

The three things the box can contain

A lift above the floor, interval clear of zero. Retargeting adds sales. Scale it into Cyber Week with a number behind it, and use the interval to set how far.

An interval that includes zero, with an MDE above the effect you expected. You learned nothing, and the correct reading is exactly that. The book is blunt about the alternative: a null result read as "does nothing" is the most common misreading in the field. Manage the channel on judgement and say so.

An interval that includes zero, with an MDE below the effect you expected. The channel is not doing what the report says. This is the outcome that pays for every other test, because the budget it releases is the only Q4 money that was ever going to move on evidence.

What to do this week

• If you own the budget: run the qualification arithmetic on retargeting alone and on retargeting plus brand search together. One of the two will clear the floor. Book that one.
• If you have to defend it: write the design down before 2 October: unit of geography, treated share, pre-period, MDE, decision rule. Registration is the cheapest defence against reinterpreting an inconvenient result in December.

The calendar has the dates.

As of 9 September 2026. The eBay and Facebook findings, the ν table and the placebo floor are from The Price of Being Found (Edition 2.10), Chapters 9, 15 and 16, with the book's scope notes.