How long should a holdout test run?

Long enough for the gap you care about to stand out from day-to-day noise. That depends on your orders a day, the smallest lift worth detecting and the share you hold out, so there is no single length that fits every store. Enter your numbers above for the minimum number of days, then round up to whole weeks.

What share of my audience or regions should I hold out?

A larger holdout shortens the test but gives up more orders while it runs; a smaller one protects revenue and takes longer. By the formula, the test is shortest at a holdout share of 1 ÷ (1 + √(1 + L)), just under half for small lifts. The planner shows the days and the orders at stake side by side, so you can weigh them.

What lift should I enter?

The smallest lift that would change your decision: if the channel caused less than this, you would stop paying for it. Work it out from what the channel costs. For example, if it costs €3,000 a month and an order leaves €20 of margin, it has to cause 150 orders a month to pay for itself. If you sell 1,650 orders a month with it on, you would sell 1,500 without it, and 150 ÷ 1,500 is a 10% lift.

What do confidence and power mean here?

Confidence is how strong the evidence must be before you call a difference real: at 95%, a channel that changes nothing still looks significant 5% of the time, in one direction or the other. Power is the chance the test catches the lift you entered if the channel really causes it: at 80%, a test of this length misses a real lift one time in five. Raising either makes the test longer.

Can I stop the test early if the result already looks significant?

No. The confidence level holds only if you look once, on an end date you set before the test starts. Checking every day and stopping at the first result that looks significant makes a false positive more likely than the level you chose. If you need to look along the way, fix the checkpoints in advance and use a sequential method built for it.

Why run a holdout when GA4 and the ad platforms already report conversions?

Reported conversions say which channel touched a sale, not whether the sale would have happened without it. A holdout answers that for one channel at a time, and costs orders while it runs. A causal read of your GA4 export estimates what every channel caused at once, from data you already have.

How long must your holdout test run?

Switch a channel off for part of your audience or regions and compare orders. Enter your numbers to see how many days the test must run before it can tell whether that channel causes sales.

Your plan

Add orders per day, the smallest lift worth detecting and the share held out to see how many days the test needs.

How it works

The planner compares two rates: orders a day where the channel still runs (the exposed group) and orders a day where it is switched off (the holdout). Daily orders are treated as Poisson counts: independent orders arriving at a steady average rate, so a count's variance equals its mean. Both rates are scaled to your whole audience, so they compare directly whatever the split.

n = (z(confidence) + z(power))²
  × [ B ÷ (1 − h) + B ÷ ((1 + L) × h) ]
  ÷ ( B × L ÷ (1 + L) )²

n: The number of days the test needs, rounded up to a whole day.
B: Orders a day today, across the whole audience or region set you will split, with the channel on. It is also the exposed group's rate, scaled to the whole audience.
L: The smallest lift worth detecting, as a fraction: 10% is 0.10. If the channel causes that lift, the holdout sells B ÷ (1 + L) a day, scaled the same way.
h: The share held out, as a fraction: 20% is 0.20. The exposed group is the other 1 − h.
z(confidence): The standard normal quantile at 1 − α/2, where α is 1 minus your confidence level: 1.960 at 95% confidence.
z(power): The standard normal quantile at your power: 0.842 at 80% power.

In words: the gap the test must see is B × L ÷ (1 + L) orders a day. Each day, the exposed group's rate carries a variance of B ÷ (1 − h) and the holdout's B ÷ ((1 + L) × h): each grows as the group's share of the audience shrinks. Over n days both variances shrink by n. The test is long enough when the gap is z(confidence) + z(power) standard errors wide: a two-sided test of statistical significance at your confidence level, with the power you chose.

The quantiles are computed in your browser with Peter Acklam's published approximation to the inverse normal distribution. When the holdout expects only a few dozen orders over the whole test, the normal approximation gets rougher and a test can fall short of the power you chose: one more reason to treat the result as a minimum.

What shortens a test

Every lever is in the formula. Twice the orders a day halves the days. A bigger lift shortens it fastest: for small lifts, halving the lift worth detecting nearly quadruples the days. A larger holdout shortens it up to a share of 1 ÷ (1 + √(1 + L)), just under half for small lifts, but gives up more orders while the test runs. Lower confidence or power shortens it too, at the cost of more wrong calls.

A worked example

Example inputs, round numbers, not a real store: 200 orders a day with the channel on everywhere, a 10% lift worth detecting, 20% of regions held out, 95% confidence, 80% power and a €60 average order value. “Fill in the worked example” in the planner enters the same numbers.

The gap to detect: 200 × 0.10 ÷ 1.10 = 18.182 orders a day.
The daily variance: 200 ÷ 0.80 + 200 ÷ (1.10 × 0.20) = 250 + 909.09 = 1,159.09.
The quantiles: (1.95996 + 0.84162)² = 2.80158² = 7.849.
The days: 7.849 × 1,159.09 ÷ 18.182² = 7.849 × 1,159.09 ÷ 330.58 = 27.52, rounded up to 28 days. That is exactly 4 full weeks.
Over those 28 days the holdout expects 28 × 0.20 × 200 ÷ 1.10 = 1,018 orders, and the exposed group 28 × 0.80 × 200 = 4,480.
If the lift is real, the holdout gives up 28 × 0.20 × (200 − 181.82) = 102 orders: about €6,109 at €60 an order, before the ad spend saved by not running the channel there.

Before you switch anything off

Treat the result as a minimum. The formula assumes independent days, no seasonality and a randomised split. Real tests with regional or weekly swings need longer.
Book whole weeks. Round the days up to a full-week multiple, so every weekday is in the test the same number of times.
Fix the end date, and do not stop early. Stopping at the first good-looking day makes a false “it works” more likely than your confidence level says. Read the result once, on the date you set.
Randomise the split. Assign users or regions to the control group at random, not by picking the quiet ones, and keep the assignment for the whole test.

Frequently asked questions

How long should a holdout test run?
Long enough for the gap you care about to stand out from day-to-day noise. That depends on your orders a day, the smallest lift worth detecting and the share you hold out, so there is no single length that fits every store. Enter your numbers above for the minimum number of days, then round up to whole weeks.
What share of my audience or regions should I hold out?
A larger holdout shortens the test but gives up more orders while it runs; a smaller one protects revenue and takes longer. By the formula, the test is shortest at a holdout share of 1 ÷ (1 + √(1 + L)), just under half for small lifts. The planner shows the days and the orders at stake side by side, so you can weigh them.
What lift should I enter?
The smallest lift that would change your decision: if the channel caused less than this, you would stop paying for it. Work it out from what the channel costs. For example, if it costs €3,000 a month and an order leaves €20 of margin, it has to cause 150 orders a month to pay for itself. If you sell 1,650 orders a month with it on, you would sell 1,500 without it, and 150 ÷ 1,500 is a 10% lift.
What do confidence and power mean here?
Confidence is how strong the evidence must be before you call a difference real: at 95%, a channel that changes nothing still looks significant 5% of the time, in one direction or the other. Power is the chance the test catches the lift you entered if the channel really causes it: at 80%, a test of this length misses a real lift one time in five. Raising either makes the test longer.
Can I stop the test early if the result already looks significant?
No. The confidence level holds only if you look once, on an end date you set before the test starts. Checking every day and stopping at the first result that looks significant makes a false positive more likely than the level you chose. If you need to look along the way, fix the checkpoints in advance and use a sequential method built for it.
Why run a holdout when GA4 and the ad platforms already report conversions?
Reported conversions say which channel touched a sale, not whether the sale would have happened without it. A holdout answers that for one channel at a time, and costs orders while it runs. A causal read of your GA4 export estimates what every channel caused at once, from data you already have.

More calculators and references on the free tools page.

A holdout tests one channel. Your GA4 export covers them all.

A holdout measures one channel at a time, and gives up orders while it runs. A causal read of your GA4 export estimates every channel at once, from data you already have.

Your GA4 export already holds how long your buyers take and which channels they touch. First finding free, in your browser; the full read is €99.

Get your first finding free All free tools

Prices exclude VAT. Full refund within 30 days, no questions asked.