A Dutch Province Geo Test Cannot Reach p Below 0.05

A geo test that treats one Dutch province and checks its result against the other eleven as placebos can tell twelve stories, the real one and eleven pretend ones. The most extreme the real result can be is best of twelve, and one in twelve is 0.083. That is above the 0.05 the world calls significance, and no budget, model or vendor changes it. If your Black Friday geo test is designed on provinces with one treated unit, its p-value was decided before launch.

This is the floor under the floor. The measurability arithmetic in Can a brand your size measure Black Friday lift at all? asks whether the test is powerful enough. This one has nothing to do with statistics and everything to do with counting, and it catches tests that cleared the first bar.

The floor by geography

With N units and one treated, permutation inference over placebo assignments yields N possible orderings, so the smallest attainable p-value is 1 divided by N. The Price of Being Found tabulates it:

The book notes the statistical literature reached the same point independently: Lei and Sudijono, in a synthetic-control methods preprint (arXiv:2401.07152v3, revised 19 April 2025, not peer-reviewed), observe that the placebo test has low resolution because its null distribution is built from only N reference estimates, which creates a barrier at common levels like 0.05 when N is small. It is a known property of the design, not a quirk of Dutch provinces.

Twelve that behave like 7.8

Dutch provinces are wildly unequal. Zuid-Holland holds 21.4% of the population, Zeeland 2.2%, and the top three are more than half the country. Weight the units by population and the effective number of regions is about 7.8, not 12. You have three regions that matter, a few that sort of matter, and a handful that barely register, and the placebo distribution is built from all of them as if they were equals.

The three fixes

1. Treat more than one unit. Two treated provinces raise the number of placebo assignments to 66 and drop the floor to 0.015, at the cost of a smaller control pool.
2. Go finer. COROP regions, of which there are 40, give a floor of 0.025 and a far more even population distribution. The book's verdict: COROP is the correct unit for Dutch geo testing and provinces are not, as a fact about arithmetic rather than about any vendor's software. Elsewhere, DMA rather than state, département rather than région.
3. Use inference that is not permutation-based, and say so plainly. Model-based intervals are not bound by 1 over N. They carry their own assumptions, and the design statement on the slide has to name them.

What you must not do is run the province design anyway and report the number that comes out, because the number was fixed before the campaign started.

Why this matters more before Black Friday than at any other time

The peak is when a test's result is read most eagerly and questioned least. A December report that says "p = 0.083, not significant" on a province design will be read as "the channel does nothing", when the design could not have said anything else. The book's sixth question for any measurement vendor is the defence: what geographic unit will you test on, and what is the minimum attainable p-value at that granularity? Ask it in September. Ask your attribution vendor for a placebo test has the companion questions.

What to do this week

• If you have to defend the number: count your units. Write 1 over N on the design document next to the MDE, and if it sits above the level you intend to report against, change the design before 2 October, not the interpretation in December.
• If you own the budget: ask whoever is designing the test which unit of geography it uses. If the answer is provinces and the treated count is one, the test cannot clear 0.05 and you have just saved eight weeks.

The calendar has the dates. A causal read on the GA4 export is not bound by the placebo floor because it is not a permutation design; it states its interval and its assumptions instead, which is the third fix above.

As of 9 September 2026. The floor table, the population weighting and the three fixes are from The Price of Being Found (Edition 2.10), Chapter 16 and Appendix D.7; the province floor of 1 in 12 is arithmetic and the book rates it Established.